DataFrame으로 데이터 조작

PySpark 입문

Ben Schmidt

Data Engineer

결측치 처리

.na.drop()으로 널 값이 있는 행 삭제

# Drop rows with any nulls
df_cleaned = df.na.drop()


# Filter out nulls
df_cleaned = df.where(col("columnName").isNotNull())

.na.fill({"column": value)로 널을 특정 값으로 대체

# Fill nulls in the age column with the value 0
df_filled = df.na.fill({"age": 0})

컬럼 작업

.withColumn()으로 계산/기존 컬럼 기반 새 컬럼 추가

# Create a new column 'age_plus_5'
df = df.withColumn("age_plus_5", df["age"] + 5)

withColumnRenamed()로 컬럼명 변경

# Rename the 'age' column to 'years'
df = df.withColumnRenamed("age", "years")

drop()으로 불필요한 컬럼 제거

# Drop the 'department' column
df = df.drop("department")

행 작업

.filter()로 조건에 맞는 행 선택

# Filter rows where salary is greater than 50000
filtered_df = df.filter(df["salary"] > 50000)

.groupBy()와 집계 함수(예: .sum(), .avg())로 요약

# Group by department and calculate the average salary
grouped_df = df.groupBy("department").avg("salary")

행 작업 결과

필터링

+------+---+-----------------+
|salary|age|      occupation |
+------+---+-----------------+
| 60000| 45|Exec-managerial  |
| 70000| 35|Prof-specialty   |
+------+---+-----------------+

GroupBy ` +----------+-----------+ |department|avg(salary)| +----------+-----------+ | HR| 80000.0| | IT| 70000.0| +----------+-----------+

치트시트

# Drop rows with any nulls
df_cleaned = df.na.drop()

#Drop nulls on a column
df_cleaned = df.where(col("columnName").isNotNull())

# Fill nulls in the age column with the value 0
df_filled = df.na.fill({"age": 0})

.withColumn()으로 계산/기존 컬럼 기반 새 컬럼 추가. 구문: .withColumn("new_col_name", "original transformation")
```
# Create a new column 'age_plus_5'
df = df.withColumn("age_plus_5", df["age"] + 5)
```
withColumnRenamed()로 컬럼명 변경 구문: withColumnRenamed(old column name,new column name`

# Rename the 'age' column to 'years'
df = df.withColumnRenamed("age", "years")

drop()으로 불필요한 컬럼 제거 구문: .drop(column name)

# Drop the 'department' column
df = df.drop("department")

# Filter rows where salary is greater than 50000
filtered_df = df.filter(df["salary"] > 50000)

.groupBy()와 집계 함수(예: .sum(), .avg())로 데이터 요약

# Group by department and calculate the average salary
grouped_df = df.groupBy("department").avg("salary")

연습해 봅시다!

PySpark 입문