Platform architecture

Introduction to Databricks Lakehouse

Gang Wang

Senior Data Scientist

The hotel analogy

Control plane - "The Front Desk"

recraft: half: A modern hotel front desk with a concierge managing bookings on screens, representing Databricks managing coordination and scheduling

Managed by Databricks
Handles coordination and requests

Data plane - "Your Hotel Room"

recraft: half: A secure private hotel room with a personal safe and workspace desk, representing the customer cloud account where data lives and compute runs

Lives in your cloud account
Where your data and compute live

The control plane

Managed by Databricks
Workspace UI and notebooks
Job scheduling and orchestration
Cluster management and configuration

recraft: half: A modern reception desk with organized monitors showing dashboards and scheduling boards, representing a control center

The data plane

recraft: half: A secure vault room with server racks and locked storage containers, representing customer-owned cloud infrastructure

Lives in your cloud account
Compute clusters run here
Data stored in your cloud storage
You control access and networking

What lives where?

nanobanana: full: Split diagram with two sections connected by arrows: left side labeled Control Plane showing Workspace UI and Job Scheduler, right side labeled Data Plane showing Clusters and Cloud Storage, arrows flowing between them

Audit logs and system tables

Audit logs track all workspace activity
Stored in your cloud account as system tables
Query with SQL for compliance reporting

SELECT user_name, action_name,
       event_time
FROM system.access.audit
WHERE action_name = 'clusterCreate'
ORDER BY event_time DESC
LIMIT 5;

Summary

Control plane - Databricks-managed (UI, scheduling, management)
Data plane - your cloud (compute, storage, data)
Audit logs - stored as system tables in your account
Your data never leaves your cloud environment

Let's practice!

Introduction to Databricks Lakehouse