Tạo và quản lý cụm

Databricks với Python SDK

Avi Steinberg

Senior Software Engineer

Hạ tầng serverless vs. managed

Serverless

Chạy trên hạ tầng do Databricks quản lý hoàn toàn
Tập trung vào code thay vì hạ tầng
Trả tiền theo nhu cầu tính toán

Managed

Kiểm soát cấu hình nhiều hơn
Hiệu quả chi phí hơn cho job chạy lâu hoặc tải dự báo

Tạo cụm Databricks Spark

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
cluster = w.clusters.create(
    cluster_name="datacamp-cluster-name",

    spark_version="latest",

    autotermination_minutes=20,

    num_workers=3,
).result()

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Liệt kê cụm

from databricks.sdk import WorkspaceClient
# Instantiate WorkspaceClient
w = WorkspaceClient()

# Print id of each cluster in workspace
clusters = w.clusters.list()
for cluster in clusters:
    print(f"ClusterId={cluster.cluster_id}")

Output:

ClusterId=0113-13328-woj98c32

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Khởi động cụm

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
cluster_id=os.environ["DATABRICKS_CLUSTER_ID"]

# Start cluster with id stored in cluster_id variable
try:
  w.clusters.start(cluster_id=cluster_id).result()
except: 
  print(f"Cannot start cluster_id={cluster_id} because it is already running")

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Kiểm tra trạng thái cụm

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()

# Print state of cluster
cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output:

cluster state=State.RUNNING

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

Xóa cụm

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
# Delete databricks cluster, with id stored in an environment variable
w.clusters.delete(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])


cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output: 
cluster state=State.TERMINATED

Ayo berlatih!

Databricks với Python SDK