Orchestration in Databricks

Databricks Concepts

Kevin Barlow

Data Analytics Practitioner

What is data orchestration?

  • Data orchestration is a form of automation!

Data Engineer

  • Enables data engineers to automate the end-to-end data life cycle
Databricks Concepts

Databricks Workflows

Databricks Workflows is a collection of built-in capabilities to orchestrate all your data processes, at no additional cost!

Example Databricks Workflow

Databricks Workflows diagram

1 https://docs.databricks.com/workflows
Databricks Concepts

What can we orchestrate?

Data engineers/data scientists

Orchestration Tasks - DE / DS

Data analysts

Orchestration Tasks - DA

Databricks Concepts

Databricks Jobs

Workflows UI

Users can create jobs directly from the Databricks UI:

  • Directly from a notebook
  • In the Workflows section

Databricks Jobs UI

1 https://docs.databricks.com/workflows/jobs
Databricks Concepts

Databricks Jobs

Programmatic

Users can also programmatically create jobs using the Jobs CLI or Jobs API with the Databricks platform.

{
  "name": "A multitask job",
  "tags": {},
  "tasks": [],
  "job_clusters": [],
  "format": "MULTI_TASK",
}
Databricks Concepts

Delta Live Tables

Delta Live Tables Diagram

Databricks Concepts

Delta Live Tables

Delta Live Tables Diagram

Databricks Concepts

Delta Live Tables

Delta Live Tables Diagram

Databricks Concepts

Let's practice!

Databricks Concepts

Preparing Video For Download...