Kerangka komputasi paralel

Pengantar Data Engineering

Vincent Vankrunkelsven

Data Engineer, DataCamp

 

Logo Apache Hadoop

Pengantar Data Engineering

HDFS

 

Diagram HDFS sebagai sistem berkas terdistribusi

Pengantar Data Engineering

MapReduce

 

Logo Hadoop MapReduce

 

Diagram contoh event Olimpiade

Pengantar Data Engineering

Hive

 

  • Berjalan di Hadoop
  • Structured Query Language: Hive SQL
  • Awalnya MapReduce, kini alat lain juga

Logo Apache Hive

Pengantar Data Engineering

Hive: contoh

 

SELECT year, AVG(age)
FROM views.athlete_events
GROUP BY year

 

Diagram Hive ke MapReduce

Pengantar Data Engineering

Gambar logo Spark

  • Hindari penulisan ke disk
  • Dikelola Apache Software Foundation
Pengantar Data Engineering

Resilient Distributed Datasets (RDD)

 

  • Spark bergantung padanya
  • Mirip daftar tuple
  • Transformasi: .map() atau .filter()
  • Aksi: .count() atau .first()
Pengantar Data Engineering

PySpark

 

  • Antarmuka Python untuk Spark
  • Abstraksi DataFrame
  • Mirip dengan Pandas
Pengantar Data Engineering

PySpark: contoh

# Load the dataset into athlete_events_spark first

(athlete_events_spark
  .groupBy('Year')
  .mean('Age')
  .show())

SELECT year, AVG(age)
FROM views.athlete_events
GROUP BY year
Pengantar Data Engineering

Ayo berlatih!

Pengantar Data Engineering

Preparing Video For Download...