Pembuatan dan pengelolaan cluster

Databricks dengan Python SDK

Avi Steinberg

Senior Software Engineer

Serverless vs. infrastruktur terkelola

Serverless

Berjalan di infrastruktur yang sepenuhnya dikelola Databricks
Fokus pada kode, bukan infrastruktur
Bayar komputasi sesuai pemakaian

Terkelola

Kontrol konfigurasi lebih besar
Lebih hemat untuk job lama atau beban prediktif

Buat cluster Databricks Spark

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
cluster = w.clusters.create(
    cluster_name="datacamp-cluster-name",

    spark_version="latest",

    autotermination_minutes=20,

    num_workers=3,
).result()

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Daftar cluster

from databricks.sdk import WorkspaceClient
# Inisialisasi WorkspaceClient
w = WorkspaceClient()

# Cetak id tiap cluster di workspace
clusters = w.clusters.list()
for cluster in clusters:
    print(f"ClusterId={cluster.cluster_id}")

Output:

ClusterId=0113-13328-woj98c32

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Mulai cluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
cluster_id=os.environ["DATABRICKS_CLUSTER_ID"]

# Mulai cluster dengan id di variabel cluster_id
try:
  w.clusters.start(cluster_id=cluster_id).result()
except: 
  print(f"Cannot start cluster_id={cluster_id} because it is already running")

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Periksa status cluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()

# Cetak status cluster
cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output:

cluster state=State.RUNNING

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Hapus cluster

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
# Hapus cluster Databricks dengan id dari variabel lingkungan
w.clusters.delete(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])


cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output: 
cluster state=State.TERMINATED

Ayo berlatih!

Databricks dengan Python SDK