Verifiability and trust

AI for Data Analysts

Andy Cotgreave

Author and Speaker, Former Technical Evangelist at Tableau

A near miss...

A near miss...

AI projects confidence

 

• Wrong outputs look as polished as right ones
• No visual signal saying "I might be wrong here"
• If you don't check, you ship the error

Verification isn't doing it twice

 

• Wrong instinct: redo the work in a BI tool
• Treat AI output like a fast junior analyst — powerful, not inherently trustworthy
• Your job: stress-test it, not redo it

The verification workflow

The verification workflow

 

1. Pick a specific data point
2. Filter the raw data manually to that data point
3. Verify the calculation for that point

The verification workflow

 

1. Copy AI's code or output
2. Paste into a different assistant or conversation
3. Ask it to find logic flaws

The verification workflow

 

1. Run a business sanity check
2. Ask: does this insight align with experience?

The verification workflow

 

1. Ask AI to run its own code against edge cases (null values, duplicates, zeros)

Match the rigor to the stakes

• Exploratory or ad-hoc work → lighter application
• Production dashboard, board presentation, etc. → heavier application

Human-in-the-loop is the job

AI sycophancy

 

• AI tends to agree with you
• Hint, push back, or imply a hypothesis → AI follows the cue
• Prompt-side defense: keep language neutral
• Verification-side defense: be hardest on findings that confirm what you wanted

Let's practice!

AI for Data Analysts