Transforming columns

Introduction to Polars

Liam Brannigan

Data Scientist & Polars Contributor

Transforming columns

Real-world data needs cleaning
Polar Expressions

Goal: prepare our data for a discounts marketing campaign

Illustration of a file and a broom to symbolize data cleaning and transpformation

Creating expressions: pl.col()

rentals.select(
  "name", 
  pl.col("price")
)

shape: (49, 2)
| name         | price |
| ---          | ---   |
| str          | i64   |
|--------------|-------|
| Waves        | 540   |
| Seashells    | 540   |
| Lake view    | 714   |
| ...          | ...   |

Arithmetic with expressions

rentals.select(
  "name",
  pl.col("price") * 0.8
)

shape: (49, 3)
| name         | price  |
| ---          | ---    |
| str          | f64    |
|--------------|--------|
| Waves        | 432.0  |
| Seashells    | 432.0  |
| Lake view    | 571.2  |
| ...          | ...    |

Chaining transformations

rentals.select(
  "name",
  (pl.col("price") * 0.8).round(0)
)

shape: (49, 2)
| name         | price  |
| ---          | ---    |
| str          | f64    |
|--------------|--------|
| Waves        | 432.0  |
| Seashells    | 432.0  |
| Lake view    | 571.0  |
| ...          | ...    |

Image of a chain.

Renaming an expression

rentals.select(
  "name", 
  (pl.col("price") * 0.8).round(0).alias("discounted_price") 
)

shape: (49, 2)
| name      | discounted_price |
| ---       | ---              |
| str       | f64              |
|-----------|------------------|
| Waves     | 432.0            |
| Seashells | 432.0            |
| Lake view | 571.0            |
| ...       | ...              |

Creating expressions from a constant

rentals.select(
  "name", 
  (pl.col("price") * 0.8).round(0).alias("discounted_price"),
  pl.lit(True).alias("available")
)

shape: (49, 3)
| name         | discounted_price | available |
| ---          | ---              | ---       |
| str          | f64              | bool      |
|--------------|------------------|-----------|
| Waves        | 432.0            | true      |
| Seashells    | 432.0            | true      |
| Lake view    | 571.0            | true      |
| ...          | ...              | ...       |

Parallel processing

Image illustrating three tasks being done in parallel

Serial processing

Image illustrating three tasks being done in sequence

Aggregating a column

rentals.select(
   "name", 
   "price",
  pl.col("price").mean().alias("avg_price"),
  pl.col("price").max().alias("max_price")
)

shape: (49, 4)
| name                | price | avg_price | max_price |
| ---                 | ---   | ---       | ---       |
| str                 | i64   | f64       | i64       |
|---------------------|-------|-----------|-----------|
| Waves               | 540   | 979.6     | 2411      |
| Seashells           | 540   | 979.6     | 2411      |
| Lake view           | 714   | 979.6     | 2411      |

Built-in transformations

rentals.select(
   "name",
   "price",
   pl.col("price").rank().alias("rank_price")
)

shape: (49, 3)
| name        | price | rank_price |
| ---         | ---   | ---        |
| str         | i64   | f64        |
|-------------|-------|------------|
| Waves       | 540   | 7.5        |
| Seashells   | 540   | 7.5        |
| Lake view   | 714   | 19.0       |
| ...         | ...   | ...        |

Dtype-specific expressions

rentals.select(
  "name",
  "type"
)

shape: (49, 2)
| name        | type    |
| ---         | ---     |
| str         | str     |
|-------------|---------|
| Waves       | Cottage |
| Seashells   | Cottage |
| ...         | ...     |

rentals.select(
  "name",
  pl.col("type").str.to_lowercase()
)

shape: (49, 2)
| name        | type    |
| ---         | ---     |
| str         | str     |
|-------------|---------|
| Waves       | cottage |
| Seashells   | cottage |
| ...         | ...     |

Let's practice!

Introduction to Polars