テキストデータのクリーニング
Data Transformation with Polars
Liam Brannigan
Data Scientist & Polars Contributor
講師の紹介
$$
$$
- リアム・ブラニガン、主任データサイエンティスト
- ML とデータエンジニアリングのスペシャリスト
- Polars コントリビューター
変換エンジン
このコースはあなた向け？
$$
- Polars DataFrame の作成
$$
- Polars の式を使う
$$
- group-by 集計を行う
第1章
第2章
第3章
第4章
データセットを見てみましょう
import polars as pl
ratings = pl.read_csv("restaurant_ratings.csv")
shape: (5, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | f64 | f64 |
|------------------|--------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5.0 | 55.0 |
| Bang Bang Burger | Forest Rd. | restaurant | 4.0 | 55.0 |
| Costa Coffee | City Point | café | 5.0 | 41.0 |
| Costa Coffee | The Moorgate | takeaway | 5.0 | 0.0 |
| The Queens Head | Denman St. | bar | 5.0 | 187.0 |
レストラン推薦アプリ
shape: (3, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | f64 | f64 |
|------------------|--------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5.0 | 55.0 |
| Bang Bang Burger | Forest Rd. | restaurant | 4.0 | 55.0 |
| Costa Coffee | City Point | café | 5.0 | 41.0 |
- 余分な空白を削除
- rating と capacity 列を変換
- 一意の識別子列を作成
式で dtype をキャストする
ratings.with_columns(
)
式で dtype をキャストする
ratings.with_columns(
pl.col("rating").cast(pl.Int64)
)
shape: (5, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | i64 | f64 |
|------------------|--------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5 | 55.0 |
| Bang Bang Burger | Forest Rd. | restaurant | 4 | 55.0 |
| Costa Coffee | City Point | café | 5 | 41.0 |
| Costa Coffee | The Moorgate | takeaway | 5 | 0.0 |
| The Queens Head | Denman St. | bar | 5 | 187.0 |
複数列のキャスト
ratings.cast( )
複数列のキャスト
ratings.cast({ })
複数列のキャスト
ratings.cast({pl.Float64: pl.Int64})
shape: (5, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | i64 | i64 |
|------------------|--------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5 | 55 |
| Bang Bang Burger | Forest Rd. | restaurant | 4 | 55 |
| Costa Coffee | City Point | café | 5 | 41 |
| Costa Coffee | The Moorgate | takeaway | 5 | 0 |
| The Queens Head | Denman St. | bar | 5 | 187 |
テキストデータのクリーニング
shape: (5, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | i64 | i64 |
|------------------|-----------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5 | 55 |
| Bang Bang Burger | Forest Rd. | restaurant | 4 | 55 |
| Costa Coffee | City Point | café | 5 | 41 |
| Costa Coffee | The Moorgate | takeaway | 5 | 0 |
| The Queens Head | Denman St. | bar | 5 | 187 |
テキストデータのクリーニング
.str.contains().str.strip_chars().str.strip_chars_start().str.to_lowercase()- ...
テキストデータのクリーニング
.str.contains().str.strip_chars().str.strip_chars_start().str.to_lowercase()- ...
1 https://docs.pola.rs/api/python/stable/reference/expressions/string.html
テキストデータのクリーニング
.str.contains().str.strip_chars().str.strip_chars_start().str.to_lowercase()- ...
1 https://docs.pola.rs/api/python/stable/reference/expressions/string.html
テキストデータのクリーニング
ratings.with_columns(
)
テキストデータのクリーニング
ratings.with_columns(
pl.col("business")
)
テキストデータのクリーニング
ratings.with_columns(
pl.col("business").str.strip_chars_start()
)
shape: (5, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | i64 | i64 |
|------------------|--------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5 | 55 |
| Bang Bang Burger | Forest Rd. | restaurant | 4 | 55 |
| Costa Coffee | City Point | café | 5 | 41 |
| Costa Coffee | The Moorgate | takeaway | 5 | 0 |
| The Queens Head | Denman St. | bar | 5 | 187 |
テキストデータの結合
shape: (5, 5)
| business | location | type | rating | capacity |
| --- | --- | --- | --- | --- |
| str | str | str | i64 | i64 |
|------------------|-----------------|------------|--------|----------|
| 7burgers | Wakey Wakey | restaurant | 5 | 55 |
| Bang Bang Burger | Forest Rd. | restaurant | 4 | 55 |
| Costa Coffee | City Point | café | 5 | 41 |
| Costa Coffee | The Moorgate | takeaway | 5 | 0 |
| The Queens Head | Denman St. | bar | 5 | 187 |
テキストデータの結合
ratings.with_columns(
)
テキストデータの結合
ratings.with_columns(
pl.concat_str(
)
テキストデータの結合
ratings.with_columns(
pl.concat_str("business", "location"
)
テキストデータの結合
ratings.with_columns(
pl.concat_str("business", "location", separator=":")
)
テキストデータの結合
ratings.with_columns(
pl.concat_str("business", "location", separator=":").alias("id")
)
shape: (5, 5)
| business | location | type | ... | id |
| --- | --- | --- | --- | --- |
| str | str | str | ... | str |
|------------------|--------------|------------|-----|-------------------------------|
| 7burgers | Wakey Wakey | restaurant | ... | 7burgers:Wakey Wakey |
| Bang Bang Burger | Forest Rd. | restaurant | ... | Bang Bang Burger:Forest Rd. |
| Costa Coffee | City Point | café | ... | Costa Coffee:City Point |
| Costa Coffee | The Moorgate | takeaway | ... | Costa Coffee:The Moorgate |
| The Queens Head | Denman St. | bar | ... | The Queens Head:Denman St. |
¡Vamos a practicar!
Data Transformation with Polars
