Skapa och hantera kluster

Databricks med Python SDK

Avi Steinberg

Senior Software Engineer

Serverlös kontra hanterad infrastruktur

Serverlös

Körs på infrastruktur som hanteras helt av Databricks
Fokus på koden, inte infrastrukturen
Betala för beräkning vid behov

Hanterad

Mer kontroll över konfigurationen
Mer kostnadseffektivt för långvariga jobb eller förutsägbara arbetsbelastningar

Skapa ett Databricks Spark-kluster

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
cluster = w.clusters.create(
    cluster_name="datacamp-cluster-name",

    spark_version="latest",

    autotermination_minutes=20,

    num_workers=3,
).result()

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Lista kluster

from databricks.sdk import WorkspaceClient
# Instantiate WorkspaceClient
w = WorkspaceClient()

# Print id of each cluster in workspace
clusters = w.clusters.list()
for cluster in clusters:
    print(f"ClusterId={cluster.cluster_id}")

Utdata:

ClusterId=0113-13328-woj98c32

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Starta ett kluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
cluster_id=os.environ["DATABRICKS_CLUSTER_ID"]

# Start cluster with id stored in cluster_id variable
try:
  w.clusters.start(cluster_id=cluster_id).result()
except: 
  print(f"Cannot start cluster_id={cluster_id} because it is already running")

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Kontrollera ett klusters tillstånd

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()

# Print state of cluster
cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Utdata:

cluster state=State.RUNNING

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Ta bort ett kluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
# Delete databricks cluster, with id stored in an environment variable
w.clusters.delete(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])


cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output: 
cluster state=State.TERMINATED

Nu kör vi en övning!

Databricks med Python SDK