Enterprise Reliability

Case Study: Sales Analytics with Databricks Genie

Stan Konkin

ML Content Developer, DataCamp

From working to trusted

A Genie space now used across a whole Sales floor, with a reliability shield over it and a VP figure reviewing it, DataCamp navy and green style

 

  • Space now serves all of Sales
  • VP wants reliability before full rollout
  • No new tables, just hardening
Case Study: Sales Analytics with Databricks Genie

Reliability is a loop

A four-step cycle, review then diagnose then curate then verify, drawn as a clean loop of arrows, DataCamp navy and green style

 

  • Review the conversations and ratings
  • Diagnose what went wrong
  • Curate the right lever
  • Verify the fix in chat
Case Study: Sales Analytics with Databricks Genie

Start by diagnosing in Monitor

$$ SCREENSHOT: The Genie Monitor tab showing a list of conversations with thumbs-up and thumbs-down ratings, filtered to the thumbs-down ones

 

$$

$$

  • Monitor tab logs real conversations
  • Filter to the thumbs-down answers
  • Read each generated SQL
Case Study: Sales Analytics with Databricks Genie

Two failures, two causes

Two icons, one a question mark with differing numbers meaning the value drifts, the other a definition crossed out meaning the term is undefined

 

  • Monthly trend drifts → loose formula
  • "Premium" undefined → Genie guesses
  • Different failures need different fixes
Case Study: Sales Analytics with Databricks Genie

Fix one, lock the trend

A monthly line chart being locked steady with a green padlock so it stops shifting, DataCamp navy and green style

 

  • Loose formula → lock with a SQL example
  • Genie reuses the exact calculation
  • Same trend, every run
Case Study: Sales Analytics with Databricks Genie

Fix two, define the term

A clear rule being added to a list of instructions, pinning down the meaning of a business term, DataCamp navy and green style

 

  • Missing definition → text instruction
  • Define the term once, precisely
  • Applied to every premium question
Case Study: Sales Analytics with Databricks Genie

Keep it reliable over time

$$

$$

SCREENSHOT: The Genie Benchmark tab listing fixed questions each paired with a ground-truth SQL answer

 

$$

$$

  • Pair questions with known-correct SQL
  • Re-run after any change
  • Catch a regression before leadership does
Case Study: Sales Analytics with Databricks Genie

What makes a space production-ready

A central dashboard with a green check, surrounded by labeled icons for benchmark, clear definitions, monitor, and locked formulas

 

  • Locked formulas, clear definitions
  • Monitored and benchmarked
  • That's how a space earns trust
Case Study: Sales Analytics with Databricks Genie

Let's practice!

Case Study: Sales Analytics with Databricks Genie

Preparing Video For Download...