Pengenalan Dask

Pemrograman Paralel dengan Dask di Python

James Fulton

Climate Informatics Researcher

Mempercepat komputasi dengan banyak inti

Komputer memiliki banyak inti
Kode harus ditulis untuk memanfaatkannya
Paket Dask dapat digunakan untuk ini
Menyelesaikan komputasi lebih cepat

Pemrograman konkuren

Diagram yang menampilkan daftar tugas dan satu jalur yang melaluinya.

Multi-threading

Daftar tugas yang dipecah menjadi dua.

Multi-threading

Dua set tugas dikirim ke dua inti CPU.

Multi-threading

Dua set tugas dijalankan dalam proses Python yang sama.

Pemrosesan paralel

Dua set tugas kini berjalan dari dua proses Python yang berbeda.

Pemrograman paralel

Multi-threading

Dua set tugas dijalankan oleh dua inti CPU dalam proses Python yang sama.

Pemrosesan paralel

Dua set tugas dijalankan oleh dua inti CPU dalam dua proses Python yang berbeda.

Evaluasi malas (lazy)

Komputasi tidak dijalankan sampai hasil dibutuhkan
Langkah untuk menghitung hasil disimpan untuk nanti
Dask membagi tugas ke thread atau proses

Dask delayed

from dask import delayed

def my_square_function(x):
    return x**2


# Create delayed version of above function
delayed_square_function = delayed(my_square_function)

Dask delayed

from dask import delayed

def my_square_function(x):
    return x**2

# Create delayed version of above function
delayed_square_function = delayed(my_square_function)

# Use the delayed function with input 4
delayed_result = delayed_square_function(4)


# Print the delayed answer
print(delayed_result)

Delayed('my_square_function-7f71b132-70a9-457a-aa52-604e8c34f8a7')

Dask delayed

from dask import delayed

def my_square_function(x):
    return x**2

# Delay and use function
delayed_result = delayed(my_square_function)(4)

print(delayed_result)

Delayed('my_square_function-7f71b132-70a9-457a-aa52-604e8c34f8a7')

Menghitung hasil

from dask import delayed

def my_square_function(x):
    return x**2

delayed_result = delayed(my_square_function)(4)

real_result = delayed_result.compute() # <- This line is where the calculation happens

# Print the answer
print(real_result)

Menggunakan operasi pada objek delayed

delayed_result1 = delayed(my_square_function)(4)

# Math operations return delayed object
delayed_result2 = (4 + delayed_result1) * 5

print(delayed_result2.compute())

Evaluasi malas (lazy)

x_list = [30, 85, 14, 12, 27, 62, 89, 15, 78,  0]

sum_of_squares = 0

for x in x_list:
    # Square and add numbers
    sum_of_squares += delayed(my_square_function)(x)

Evaluasi malas (lazy)

x_list = [30, 85, 14, 12, 27, 62, 89, 15, 78,  0]

sum_of_squares = 0

for x in x_list:
    # Square and add numbers
    sum_of_squares += delayed(my_square_function)(x)

result = sum_of_squares.compute()

# Print the answer
print(result)

Berbagi komputasi

delayed_intermediate = delayed(my_square_function)(3)

# These two results both use delayed_intermediate
delayed_result1 = delayed_intermediate - 5
delayed_result2 = delayed_intermediate + 4

# delayed_3_squared will be computed twice
print('delayed_result1:', delayed_result1.compute())
print('delayed_result2:', delayed_result2.compute())

delayed_result1: 4
delayed_result2: 13

Berbagi komputasi

import dask

# delayed_intermediate will be computed once
comp_result1, comp_result2 = dask.compute(delayed_result1, delayed_result2)

print('comp_result1:', comp_result1)
print('comp_result2:', comp_result2)

delayed_result1: 4
delayed_result2: 13

Ayo berlatih!

Pemrograman Paralel dengan Dask di Python