Création et gestion de jobs

Databricks avec le SDK Python

Avi Steinberg

Senior Software Engineer

Chemin d'un Notebook Databricks

Les jobs Databricks peuvent exécuter du code dans un Notebook Databricks
WorkspaceClient.current_user.me().user_name récupère le nom d'utilisateur connecté
Le chemin d'un notebook est /Users/${username}/${notebook_name}

# Supposons qu'un notebook nommé My_Notebook existe dans l'espace de travail
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/My_Notebook'

Créer un job Databricks

WorkspaceClient.jobs.create(createParams)

Renseignez les paramètres name et tasks pour décrire le job créé

createParams:
  name: str
  tasks: List[Task]
Task:
  description=str
  notebook_task=NotebookTask
  task_key=str
NotebookTask:
  notebook_path: str

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Créer et exécuter un job Databricks

from databricks.sdk import WorkspaceClient

# Create notebook path pointing to notebook called "My_Notebook"
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/My_Notebook'


# Create a job that runs the Datacamp_Test_Notebook
new_job = w.jobs.create(name='sdk-dc-project-task',
                            tasks=[jobs.Task(
                              description="create_notebook_test",
                              notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
                              task_key="my-key")
                            ])
print(f"New Job Id={new_job.job_id}")

w.jobs.run_now(job_id=new_job.job_id).result() # Run created Job

¹ https://docs.databricks.com/en/dev-tools/sdk-python.html

Lister les jobs Databricks

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
jobs = w.jobs.list()
for job in jobs:
    print(f"JobId={job.job_id}")

JobId=888763141802192
JobId=37050453972815
JobId=681550316180975
JobId=629994089852037

Suppression d'un job Databricks

WorkspaceClient.jobs.delete(job_id: str)

¹ 1. https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Suppression d'un job Databricks

# Pre-requisite to this is to create desired 
# Databricks notebook in databricks cluster
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/My_Notebook'
# Create a job that runs the Datacamp_Test_Notebook
new_job = w.jobs.create(name='sdk-dc-project-task',
                            tasks=[
                              jobs.Task(description="create_notebook_test",
                                        existing_cluster_id=cluster_id,
                      notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
                                        task_key="my-key")
                            ])

w.jobs.delete(job_id=new_job.job_id)

¹ 1. https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Syntaxe cron

Expression cron pour planifier à :

03:30:00 chaque jour = 0 30 3 * * ?
14:45:00 chaque jour = 0 45 2 * * ?

¹ https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html

Planifier un job

Nous pouvons planifier l'exécution d'un notebook chaque jour à 3 h

# Create a job that runs the Datacamp_Test_Notebook
cron_expression = "0 0 3 * * ?"

created_job = w.jobs.create(
                name='sdk-dc-project-task',
                tasks=[jobs.Task(description="test",

                    notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
                        task_key="my-key")],
                timeout_seconds=3600,

                  schedule=jobs.CronSchedule(quartz_cron_expression=cron_expression,
                                           timezone_id="America/New_York")
               )

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Passons à la pratique !

Databricks avec le SDK Python