Benchmarking and continuous improvement

Introduction to Databricks Genie

Gang Wang

Senior Data Scientist

What is benchmarking

Gold Standard questions and answers

• Calculates accuracy

Benchmarking frequency

• 10-20 Gold Standard questions covering simple counts to complex Trusted KPIs
• Run during development - daily or after every significant curation change
• After schema changes run the full suite immediately when adding new tables or columns
• In production - run weekly or monthly as a health check

Interpreting results

• Pass rate gives you a headline score
• Real value is in interpreting failures
• "Total sales by city" passes, but "Total revenue by municipality" fails?
• That means Genie needs Synonyms, not a logic change
• Compare Expected SQL vs Generated SQL to find the gap

Prioritizing fixes

• Identify failure patterns in your benchmark results
• Jump from result to the Curation menu to add synonyms and examples
• Prioritize by frequency and business impact
• Fix high-frequency, high-impact failures first

Continuous improvement cycle

$$

Step 1: Build your test suite of Gold Standard questions.

Continuous improvement cycle

$$

Step 2: Execute and interpret results. What failed and why?

Continuous improvement cycle

$$

Step 3: Prioritize and curate fixes using synonyms, relationships, or SQL examples.

Continuous improvement cycle

$$

Step 4: Verify improvement by re-running the benchmark. Confirm accuracy went up before deploying.

Scaling across the organization

• Centralize data and permissions with Unity Catalog
• Use Trusted Assets for company-wide KPIs
• Reserve Editor access for trained Data Stewards
• Mandate thumbs-up/thumbs-down feedback
• Run regular benchmarks to catch accuracy drift

Let's practice!

Introduction to Databricks Genie