Trappole con le categoriche

Lavorare con i dati categorici in Python

Kasey Jones

Research Data Scientist

Auto usate: dataset finale

import pandas as pd

used_cars = pd.read_csv("used_cars.csv")
used_cars.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 38531 entries, 0 to 38530
Data columns (total 30 columns):
 #   Column             Non-Null Count  Dtype  
 --  ------             --------------  -----  
 0   manufacturer_name  38531 non-null  object 
 1   model_name         38531 non-null  object 
 2   transmission       38531 non-null  object  
...

Grandi risparmi di memoria

used_cars['manufacturer_name'].describe()

count          38531
unique            55
top       Volkswagen
freq            4243
Name: manufacturer_name, dtype: object

print("As object: ", used_cars['manufacturer_name'].nbytes)
print("As category: ", used_cars['manufacturer_name'].astype('category').nbytes)

As object: 308248
As category: 38971

¹ https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html

Pochi risparmi di memoria

used_cars['odometer_value'].astype('object').describe()

count      38531
unique      6063
top       300000
freq        1794
Name: odometer_value, dtype: int64

print(f"As float: {used_cars['odometer_value'].nbytes}")
print(f"As category: {used_cars['odometer_value'].astype('category').nbytes}")

As float: 308248
As category: 125566

Le categorie possono frustrare

Usare l’accessor .str per manipolare i dati converte la Series in object.
Il metodo .apply() produce una nuova Series di tipo object.
Le operazioni comuni (aggiungere, rimuovere, sostituire, impostare categorie) non gestiscono allo stesso modo le categorie mancanti.
Le funzioni NumPy in genere non funzionano con Series categoriche.

Verifica e converti

Verifica

used_cars["color"] = used_cars["color"].astype("category")
used_cars["color"] = used_cars["color"].str.upper()
print(used_cars["color"].dtype)

object

Converti

used_cars["color"] = used_cars["color"].astype("category")
print(used_cars["color"].dtype)

category

Cerca valori mancanti

Imposta categorie

used_cars["color"] = used_cars["color"].astype("category")
used_cars["color"].cat.set_categories(["black", "silver", "blue"], inplace=True)

used_cars["color"].value_counts(dropna=False)

NaN       18172
black      7705
silver     6852
blue       5802
Name: color, dtype: int64

Usare array NumPy

used_cars['number_of_photos'] = used_cars['number_of_photos'].astype("category")
used_cars['number_of_photos'].sum()  # <--- Gives an Error

TypeError: Categorical cannot perform the operation sum

used_cars['number_of_photos'].astype(int).sum()

Nota:

# .str converts the column to an array
used_cars["color"].str.contains("red")

0        False
1        False
...

Esercizio sulle trappole

Lavorare con i dati categorici in Python