From manual to automated deployment

Introduction to Databricks Lakehouse

Gang Wang

Senior Data Scientist

The manual deployment problem

$$

  • Configure jobs through the UI, one at a time
  • Copy notebooks to production folders manually
  • Settings drift between staging and production
  • No way to reproduce a deployment from scratch

$$

recraft: half: A person at a cluttered desk covered in sticky notes and mismatched papers, representing the chaos of manual deployment processes

Introduction to Databricks Lakehouse

Why this matters at scale

$$

  • A typical project has jobs, pipelines, clusters, permissions, and schedules
  • Each configured separately in the UI
  • One team member leaves - their configuration knowledge leaves with them
  • An audit asks "what changed last month?" - no clear answer

$$

recraft: half: An office whiteboard covered in tangled diagrams, crossed-out notes, and question marks, representing lost institutional knowledge and configuration chaos

Introduction to Databricks Lakehouse

What are Databricks Asset Bundles?

$$

radial: Databricks Asset Bundles

$$

  • Everything in a project folder, versioned in Git, deployed with a single CLI command
Introduction to Databricks Lakehouse

DAB advantages

$$

  • Reproducible - deploy the same config every time
  • Version-controlled - changes tracked in Git
  • Environment promotion - dev → staging → prod
  • CI/CD integration - automate deployments in pipelines

$$

recraft: half: A factory assembly line with identical boxes being produced one after another on a conveyor belt, representing reproducible automated deployment

Introduction to Databricks Lakehouse

Traditional vs. DAB

$$

Traditional DAB
Configuration UI clicks YAML files in Git
Reproducibility Manual, error-prone Automated, consistent
Environment promotion Copy and reconfigure bundle deploy --target prod
Rollback Recreate from memory git revert + redeploy
Collaboration One person at a time Full Git workflow

Introduction to Databricks Lakehouse

Summary

$$

  • Manual UI deployment is fragile and doesn't scale
  • Databricks Asset Bundles describe your project in YAML
  • Benefits: reproducibility, version control, CI/CD, environment promotion
  • Everything lives in Git - one source of truth for your deployment
Introduction to Databricks Lakehouse

Let's practice!

Introduction to Databricks Lakehouse

Preparing Video For Download...