使用 Categorical 和 Enum 数据类型
使用 Polars 扩展与优化数据流水线
Liam Brannigan
Data Scientist & Polars Contributor
对重复字符串编码
对重复字符串编码
含重复标签的数据集
events = pl.read_parquet("chicago_events.parquet")
含重复标签的数据集
events["area"].value_counts(sort=True).head(5)
shape: (5, 2)
| area | count |
| --- | --- |
| str | u32 |
|------------- |-------|
| Loop | 35982 |
| West Loop | 27432 |
| Andersonville | 24660 |
| Downtown | 24320 |
| Pullman | 21985 |
创建 Categorical 列
events_cat = events.with_columns(
pl.col("area").cast(pl.Categorical)
)
创建 Categorical 列
events_cat.select("event_title","area").head()
shape: (5, 2)
| event_title | area |
| --- | --- |
| str | cat |
|--------------------|---------------|
| Folk Festival | Andersonville |
| Fireworks Night | Downtown |
| Greektown Market | West Side |
| Rail History Day | Pullman |
| Grant Park Concert | Loop |
创建 Categorical 列
events_cat.select("event_title","area").with_columns(
pl.col("area").to_physical().alias("physical")
).head(3)
shape: (3, 3)
| event_title | area | physical |
| --- | --- | --- |
| str | cat | u32 |
|--------------------|---------------|----------|
| Folk Festival | Andersonville | 0 |
| Fireworks Night | Downtown | 1 |
| Greektown Market | West Side | 2 |
使用分类表达式
events_cat.select("event_title","area",
pl.col("area").cat.starts_with("West").alias("westside"),
)
使用分类表达式
events_cat.select("event_title","area",
pl.col("area").cat.starts_with("West").alias("westside"),
).filter(pl.col("westside")).head(2)
shape: (2, 3)
| event_title | area | westside |
| --- | --- | --- |
| str | cat | bool |
|-------------------------|-----------|----------|
| Greektown Market | West Side | true |
| West Loop Chef Showcase | West Loop | true |
- 其他字符串操作 → 转回为 String
Categorical 与 Enum
创建 Enum 列
area_enum = pl.Enum([
"Albany Park",
"Andersonville",
...
"West Loop",
"West Side",
"Wicker Park",
])
创建 Enum 列
events_enum = events.with_columns(
pl.col("area").cast(area_enum)
)
创建 Enum 列
events_enum.select("event_title", "area").head()
shape: (5, 2)
| event_title | area |
| --- | --- |
| str | enum |
|------------------------------|----------------|
| Greektown Market | West Side |
| Folk Festival | Andersonville |
| West Loop Chef Showcase | West Loop |
| Rail History Day | Pullman |
| Fireworks Night | Downtown |
查看 Enum 数据类型
events_enum.schema["area"]
Enum(categories=['Albany Park', 'Andersonville', 'Chinatown', 'Downtown', ...])
- Enum 适用于固定词表
- Categorical 适用于可变词表
Enum 的内存效率
events_enum.select("event_title","area").with_columns(
pl.col("area").to_physical().alias("physical")
).head(3)
shape: (3, 3)
| event_title | area | physical |
| --- | --- | --- |
| str | enum | u8 |
|--------------------|---------------|----------|
| Folk Festival | Andersonville | 1 |
| Fireworks Night | Downtown | 3 |
| Greektown Market | West Side | 16 |
让我们练习！
使用 Polars 扩展与优化数据流水线
