Cadre de calcul paralel

Introducere în Data Engineering

Vincent Vankrunkelsven

Data Engineer @ DataCamp

Sigla Apache Hadoop

HDFS

Diagramă HDFS ca sistem de fișiere distribuit

Sigla Hadoop MapReduce

Diagramă exemplu evenimente olimpice

Sigla Apache Hive

SELECT year, AVG(age)
FROM views.athlete_events
GROUP BY year

Diagramă Hive la MapReduce

Sigla Spark

# Load the dataset into athlete_events_spark first

(athlete_events_spark
  .groupBy('Year')
  .mean('Age')
  .show())

SELECT year, AVG(age)
FROM views.athlete_events
GROUP BY year

Introducere în Data Engineering