क्लस्टर बनाना और प्रबंधन

Python SDK के साथ Databricks

Avi Steinberg

Senior Software Engineer

Serverless बनाम managed इंफ्रास्ट्रक्चर

Serverless

Databricks द्वारा पूरी तरह मैनेज्ड इंफ्रा पर चलता है
इंफ्रास्ट्रक्चर की जगह कोड पर फोकस करें
ऑन-डिमांड कंप्यूट का भुगतान करें

Managed

कॉन्फ़िगरेशन पर ज्यादा नियंत्रण
लंबे चलने वाले जॉब्स या प्रेडिक्टिव वर्कलोड्स के लिए अधिक किफायती

Databricks Spark क्लस्टर बनाएँ

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
cluster = w.clusters.create(
    cluster_name="datacamp-cluster-name",

    spark_version="latest",

    autotermination_minutes=20,

    num_workers=3,
).result()

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

क्लस्टर्स की सूची

from databricks.sdk import WorkspaceClient
# Instantiate WorkspaceClient
w = WorkspaceClient()

# Print id of each cluster in workspace
clusters = w.clusters.list()
for cluster in clusters:
    print(f"ClusterId={cluster.cluster_id}")

Output:

ClusterId=0113-13328-woj98c32

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

क्लस्टर शुरू करें

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
cluster_id=os.environ["DATABRICKS_CLUSTER_ID"]

# Start cluster with id stored in cluster_id variable
try:
  w.clusters.start(cluster_id=cluster_id).result()
except: 
  print(f"Cannot start cluster_id={cluster_id} because it is already running")

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

क्लस्टर की स्थिति जाँचें

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()

# Print state of cluster
cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output:

cluster state=State.RUNNING

¹ https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

क्लस्टर हटाएँ

from databricks.sdk import WorkspaceClient
import os

w = WorkspaceClient()
# Delete databricks cluster, with id stored in an environment variable
w.clusters.delete(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])


cluster_info = w.clusters.get(cluster_id=os.environ["DATABRICKS_CLUSTER_ID"])
print(f"cluster state={cluster_info.state}")

Output: 
cluster state=State.TERMINATED

अभ्यास करते हैं!

Python SDK के साथ Databricks