Percentielen en histogrammen berekenen

Data transformatie met Polars

Liam Brannigan

Data Scientist & Polars Contributor

Percentielen en histogrammen

reviews = pl.read_csv("restaurants_quantiles.csv")

shape: (15, 4)
| business         | location      | review | price |
| ---              | ---           | ---    | ---   |
| str              | str           | f64    | i64   |
|------------------|---------------|--------|-------|
| 7burgers         | Wakey Wakey   | 4.2    | 15    |
| Bang Bang Burger | Forest Rd.    | 3.8    | 12    |
| ...              | ...           | ...    | ...   |
| The Old Ivy      | Angel         | 4.1    | 28    |
| Franco Manca     | Brixton Rd.   | 4.0    | 22    |

Categoriseren op percentielen

Staafdiagram met restaurantprijzen in oplopende volgorde.

Categoriseren op percentielen

Staafdiagram met restaurantprijzen in oplopende volgorde met de 33e percentiel als referentielijn.

Categoriseren op percentielen

Staafdiagram met restaurantprijzen in oplopende volgorde met de 33e en 67e percentielen als referentielijnen.

Reviews-dataset

reviews = pl.read_csv("restaurants_quantiles.csv")

shape: (15, 4)
| business         | location      | review | price |
| ---              | ---           | ---    | ---   |
| str              | str           | f64    | i64   |
|------------------|---------------|--------|-------|
| 7burgers         | Wakey Wakey   | 4.2    | 15    |
| Bang Bang Burger | Forest Rd.    | 3.8    | 12    |
| ...              | ...           | ...    | ...   |
| The Old Ivy      | Angel         | 4.1    | 28    |
| Franco Manca     | Brixton Rd.   | 4.0    | 22    |

Percentielen berekenen

reviews.select(


)

Percentielen berekenen

reviews.select(
    pl.col("price")

)

Percentielen berekenen

reviews.select(
    pl.col("price").quantile(0.33)

)

Percentielen berekenen

reviews.select(
    pl.col("price").quantile(0.33).alias("budget_cutoff"),

)

Percentielen berekenen

reviews.select(
    pl.col("price").quantile(0.33).alias("budget_cutoff"),
    pl.col("price").quantile(0.67).alias("mid_cutoff")
)

shape: (1, 2)
| budget_cutoff | mid_cutoff |
| ---           | ---        |
| f64           | f64        |
|---------------|------------|
| 14.0          | 20.0       |

Percentielen gebruiken voor categorieën

reviews.with_columns(
    pl.when(pl.col("price") <= pl.col("price").quantile(0.33)).then(pl.lit("budget"))


)

Percentielen gebruiken voor categorieën

reviews.with_columns(
    pl.when(pl.col("price") <= pl.col("price").quantile(0.33)).then(pl.lit("budget"))
    .when(pl.col("price") <= pl.col("price").quantile(0.67)).then(pl.lit("mid"))

)

Percentielen gebruiken voor categorieën

reviews.with_columns(
    pl.when(pl.col("price") <= pl.col("price").quantile(0.33)).then(pl.lit("budget"))
    .when(pl.col("price") <= pl.col("price").quantile(0.67)).then(pl.lit("mid"))
    .otherwise(pl.lit("premium")).alias("price_band")
)

shape: (15, 5)
| business         | location      | review | price | price_band |
| ---              | ---           | ---    | ---   | ---        |
| str              | str           | f64    | i64   | str        |
|------------------|---------------|--------|-------|------------|
| 7burgers         | Wakey Wakey   | 4.2    | 15    | mid        |
| Bang Bang Burger | Forest Rd.    | 3.8    | 12    | budget     |
| ...              | ...           | ...    | ...   | ...        |

Percentiel- vs. vaste intervallen

Percentieldrempels

Gebaseerd op huidige data
Gebalanceerde groepsgroottes
Past zich aan bij veranderende data

Vaste drempels

Voorbeeld: 10 en 20 pond
Onbekende groepsgroottes
Makkelijk te begrijpen

Categorieën met vaste drempels

reviews.with_columns(


)

Categorieën met vaste drempels

reviews.with_columns(
    pl.col("price")

)

Categorieën met vaste drempels

reviews.with_columns(
    pl.col("price")
    .cut(
)

Categorieën met vaste drempels

reviews.with_columns(
    pl.col("price")
    .cut(breaks=[10, 20] 
)

Categorieën met vaste drempels

reviews.with_columns(
    pl.col("price")
    .cut(breaks=[10, 20], labels=["budget", "mid", "premium"]) 
)

Categorieën met vaste drempels

reviews.with_columns(
    pl.col("price")
    .cut(breaks=[10, 20], labels=["budget", "mid", "premium"]).alias("price_band")
)

shape: (15, 5)
| business         | location      | review | price | price_band |
| ---              | ---           | ---    | ---   | ---        |
| str              | str           | f64    | i64   | cat        |
|------------------|---------------|--------|-------|------------|
| 7burgers         | Wakey Wakey   | 4.2    | 15    | mid        |
| Costa Coffee     | City Point    | 4.5    | 8     | budget     |
| The Queens Head  | Denman St.    | 4.7    | 25    | premium    |
| ...              | ...           | ...    | ...   | ...        |

Een histogram bouwen

reviews["price"]

Een histogram bouwen

reviews["price"].hist()

shape: (10, 3)
| breakpoint | category     | count |
| ---        | ---          | ---   |
| f64        | cat          | u32   |
|------------|--------------|-------|
| 8.4        | [6.0, 8.4]   | 3     |
| 10.8       | (8.4, 10.8]  | 1     |
| ...        | ...          | ...   |
| 27.6       | (25.2, 27.6] | 0     |
| 30.0       | (27.6, 30.0] | 2     |

Histogram met vast aantal bins bouwen

reviews["price"].hist(bin_count=5)

shape: (5, 3)
| breakpoint | category      | count |
| ---        | ---           | ---   |
| f64        | cat           | u32   |
|------------|---------------|-------|
| 10.8       | [6.0, 10.8]   | 4     |
| 15.6       | (10.8, 15.6]  | 3     |
| 20.4       | (15.6, 20.4]  | 3     |
| 25.2       | (20.4, 25.2]  | 3     |
| 30.0       | (25.2, 30.0]  | 2     |

Histogram met vaste breekpunten bouwen

reviews["price"].hist(bins=[0, 10, 20, 100])

shape: (3, 3)
| breakpoint | category       | count |
| ---        | ---            | ---   |
| f64        | cat            | u32   |
|------------|----------------|-------|
| 10.0       | [0.0, 10.0]    | 4     |
| 20.0       | (10.0, 20.0]   | 6     |
| 100.0      | (20.0, 100.0]  | 5     |

Het histogram plotten

import plotly.express as px


hist = reviews["price"].hist(bins=[0, 10, 20, 100])

Het histogram plotten

import plotly.express as px

hist = reviews["price"].hist(bins=[0, 10, 20, 100])

fig = px.bar(hist, x="category", y="count")

fig.show()

Het histogram plotten

Laten we oefenen!

Data transformatie met Polars