Creación y gestión de jobs

Databricks con el SDK de Python

Avi Steinberg

Senior Software Engineer

Ruta de Notebook en Databricks

Los jobs de Databricks pueden ejecutar código en un Notebook de Databricks
WorkspaceClient.current_user.me().user_name obtiene el usuario conectado
La ruta de un notebook es /Users/${username}/${notebook_name}

# Supón que hay un notebook en el workspace llamado My_Notebook
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/My_Notebook'

Crear un job de Databricks

WorkspaceClient.jobs.create(createParams)

Pasa los parámetros name y tasks para describir el job que se crea

createParams:
  name: str
  tasks: List[Task]
Task:
  description=str
  notebook_task=NotebookTask
  task_key=str
NotebookTask:
  notebook_path: str

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Crear y ejecutar un job de Databricks

from databricks.sdk import WorkspaceClient

# Crea la ruta a un notebook llamado "My_Notebook"
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/My_Notebook'


# Crea un job que ejecuta Datacamp_Test_Notebook
new_job = w.jobs.create(name='sdk-dc-project-task',
                            tasks=[jobs.Task(
                              description="create_notebook_test",
                              notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
                              task_key="my-key")
                            ])
print(f"New Job Id={new_job.job_id}")

w.jobs.run_now(job_id=new_job.job_id).result() # Ejecuta el job creado

¹ https://docs.databricks.com/en/dev-tools/sdk-python.html

Listar jobs de Databricks

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
jobs = w.jobs.list()
for job in jobs:
    print(f"JobId={job.job_id}")

JobId=888763141802192
JobId=37050453972815
JobId=681550316180975
JobId=629994089852037

Eliminar un job de Databricks

WorkspaceClient.jobs.delete(job_id: str)

¹ 1. https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Eliminar un job de Databricks

# Requisito previo: crear el notebook deseado
# de Databricks en el clúster de Databricks
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/My_Notebook'
# Crea un job que ejecuta Datacamp_Test_Notebook
new_job = w.jobs.create(name='sdk-dc-project-task',
                            tasks=[
                              jobs.Task(description="create_notebook_test",
                                        existing_cluster_id=cluster_id,
                      notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
                                        task_key="my-key")
                            ])

w.jobs.delete(job_id=new_job.job_id)

¹ 1. https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

Sintaxis cron

Expresiones cron para programar a:

3:30:00 a. m. todos los días = 0 30 3 * * ?
2:45:00 p. m. todos los días = 0 45 2 * * ?

¹ https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html

Programar un job

Podemos programar un job para ejecutar el notebook cada día a las 3 a. m.

# Crea un job que ejecuta Datacamp_Test_Notebook
cron_expression = "0 0 3 * * ?"

created_job = w.jobs.create(
                name='sdk-dc-project-task',
                tasks=[jobs.Task(description="test",

                    notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
                        task_key="my-key")],
                timeout_seconds=3600,

                  schedule=jobs.CronSchedule(quartz_cron_expression=cron_expression,
                                           timezone_id="America/New_York")
               )

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

¡Vamos a practicar!

Databricks con el SDK de Python