Orchestration in Databricks

Concetti di Databricks

Kevin Barlow

Data Analytics Practitioner

What is data orchestration?

  • Data orchestration is a form of automation!

Data Engineer

  • Enables data engineers to automate the end-to-end data life cycle
Concetti di Databricks

Databricks Workflows

Databricks Workflows is a collection of built-in capabilities to orchestrate all your data processes, at no additional cost!

Example Databricks Workflow

Databricks Workflows diagram

1 https://docs.databricks.com/workflows
Concetti di Databricks

What can we orchestrate?

Data engineers/data scientists

Orchestration Tasks - DE / DS

Data analysts

Orchestration Tasks - DA

Concetti di Databricks

Databricks Jobs

Workflows UI

Users can create jobs directly from the Databricks UI:

  • Directly from a notebook
  • In the Workflows section

Databricks Jobs UI

1 https://docs.databricks.com/workflows/jobs
Concetti di Databricks

Databricks Jobs

Programmatic

Users can also programmatically create jobs using the Jobs CLI or Jobs API with the Databricks platform.

{
  "name": "A multitask job",
  "tags": {},
  "tasks": [],
  "job_clusters": [],
  "format": "MULTI_TASK",
}
Concetti di Databricks

Delta Live Tables

Delta Live Tables Diagram

Concetti di Databricks

Delta Live Tables

Delta Live Tables Diagram

Concetti di Databricks

Delta Live Tables

Delta Live Tables Diagram

Concetti di Databricks

Let's practice!

Concetti di Databricks

Preparing Video For Download...