集群创建与管理

使用 Python SDK 的 Databricks

Avi Steinberg

Senior Software Engineer

无服务器 vs. 托管型基础设施

无服务器

运行在 Databricks 全托管基础设施上
专注代码而非基础设施
按需为计算付费

托管

配置控制更高
长时作业或可预测负载更省成本

创建 Databricks Spark 集群

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
cluster = w.clusters.create(
    cluster_name="datacamp-cluster-name",

    spark_version="latest",

    autotermination_minutes=20,

    num_workers=3,
).result()

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

列出集群

from databricks.sdk import WorkspaceClient
# 实例化 WorkspaceClient
w = WorkspaceClient()

# 打印工作区中每个集群的 ID
clusters = w.clusters.list()
for cluster in clusters:
    print(f"ClusterId={cluster.cluster_id}")

输出：

ClusterId=0113-13328-woj98c32

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

启动集群

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
cluster_id=os.environ["DATABRICKS_CLUSTER_ID"]

# 使用 cluster_id 变量中的 ID 启动集群
try:
  w.clusters.start(cluster_id=cluster_id).result()
except: 
  print(f"Cannot start cluster_id={cluster_id} because it is already running")

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

检查集群状态

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()

# 打印集群状态
cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

输出：

cluster state=State.RUNNING

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

删除集群

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
# 删除 Databricks 集群（ID 存于环境变量）
w.clusters.delete(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])


cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

输出：
cluster state=State.TERMINATED

Passons à la pratique !

使用 Python SDK 的 Databricks