Criação e gestão de clusters

Databricks com o SDK em Python

Avi Steinberg

Senior Software Engineer

Infraestrutura serverless vs. gerenciada

Serverless

Roda em infraestrutura totalmente gerenciada pela Databricks
Foque no código, não na infraestrutura
Pague pelo compute sob demanda

Gerenciada

Mais controle da configuração
Mais econômica para jobs longos ou cargas previsíveis

Criar um cluster Spark no Databricks

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
cluster = w.clusters.create(
    cluster_name="datacamp-cluster-name",

    spark_version="latest",

    autotermination_minutes=20,

    num_workers=3,
).result()

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Listar clusters

from databricks.sdk import WorkspaceClient
# Instancia o WorkspaceClient
w = WorkspaceClient()

# Imprime o id de cada cluster no workspace
clusters = w.clusters.list()
for cluster in clusters:
    print(f"ClusterId={cluster.cluster_id}")

Output:

ClusterId=0113-13328-woj98c32

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Iniciar um cluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
cluster_id=os.environ["DATABRICKS_CLUSTER_ID"]

# Inicia o cluster com o id em cluster_id
try:
  w.clusters.start(cluster_id=cluster_id).result()
except: 
  print(f"Cannot start cluster_id={cluster_id} because it is already running")

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Verificar o estado de um cluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()

# Imprime o estado do cluster
cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output:

cluster state=State.RUNNING

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Excluir um cluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
# Exclui o cluster do Databricks com id no variável de ambiente
w.clusters.delete(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])


cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output: 
cluster state=State.TERMINATED

Vamos praticar!

Databricks com o SDK em Python