使用 DataFrame 進行資料操作
PySpark 入門
Ben Schmidt
Data Engineer
處理遺漏值
- 使用
.na.drop()移除含 null 的列
# Drop rows with any nulls df_cleaned = df.na.drop()# Filter out nulls df_cleaned = df.where(col("columnName").isNotNull())
- 使用
.na.fill({"column": value)以特定值取代 null
# Fill nulls in the age column with the value 0
df_filled = df.na.fill({"age": 0})
欄位操作
- 使用
.withColumn()依計算或既有欄位新增欄位
# Create a new column 'age_plus_5'
df = df.withColumn("age_plus_5", df["age"] + 5)
- 使用
withColumnRenamed()重新命名欄位
# Rename the 'age' column to 'years'
df = df.withColumnRenamed("age", "years")
- 使用
drop()移除不需要的欄位
# Drop the 'department' column
df = df.drop("department")
列操作
- 使用
.filter()依條件篩選列
# Filter rows where salary is greater than 50000
filtered_df = df.filter(df["salary"] > 50000)
- 使用
.groupBy()與彙總函式（如.sum()、.avg()）摘要資料
# Group by department and calculate the average salary
grouped_df = df.groupBy("department").avg("salary")
列操作結果
篩選
+------+---+-----------------+ |salary|age| occupation | +------+---+-----------------+ | 60000| 45|Exec-managerial | | 70000| 35|Prof-specialty | +------+---+-----------------+群組彙總
`+----------+-----------+ |department|avg(salary)| +----------+-----------+ | HR| 80000.0| | IT| 70000.0| +----------+-----------+
`
小抄
# Drop rows with any nulls df_cleaned = df.na.drop()#Drop nulls on a column df_cleaned = df.where(col("columnName").isNotNull())# Fill nulls in the age column with the value 0 df_filled = df.na.fill({"age": 0})
使用
.withColumn()依計算或既有欄位新增欄位。語法：.withColumn("new_col_name", "original transformation")# Create a new column 'age_plus_5' df = df.withColumn("age_plus_5", df["age"] + 5)使用
withColumnRenamed()重新命名欄位 語法：withColumnRenamed(old column name,new column name`
# Rename the 'age' column to 'years'
df = df.withColumnRenamed("age", "years")
- 使用
drop()移除不需要的欄位 語法：.drop(column name)
# Drop the 'department' column
df = df.drop("department")
# Filter rows where salary is greater than 50000
filtered_df = df.filter(df["salary"] > 50000)
- 使用
.groupBy()與彙總函式（如.sum()、.avg()）摘要資料
# Group by department and calculate the average salary
grouped_df = df.groupBy("department").avg("salary")
一起來練習吧！
PySpark 入門
