Fallgropar med kategorier

Arbeta med kategoriska data i Python

Kasey Jones

Research Data Scientist

Begagnade bilar: den slutliga datamängden

import pandas as pd

used_cars = pd.read_csv("used_cars.csv")
used_cars.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 38531 entries, 0 to 38530
Data columns (total 30 columns):
 #   Column             Non-Null Count  Dtype  
 --  ------             --------------  -----  
 0   manufacturer_name  38531 non-null  object 
 1   model_name         38531 non-null  object 
 2   transmission       38531 non-null  object  
...

Stora minnesbesparingar

used_cars['manufacturer_name'].describe()

count          38531
unique            55
top       Volkswagen
freq            4243
Name: manufacturer_name, dtype: object

print("As object: ", used_cars['manufacturer_name'].nbytes)
print("As category: ", used_cars['manufacturer_name'].astype('category').nbytes)

As object: 308248
As category: 38971

¹ https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html

Små minnesbesparingar

used_cars['odometer_value'].astype('object').describe()

count      38531
unique      6063
top       300000
freq        1794
Name: odometer_value, dtype: int64

print(f"As float: {used_cars['odometer_value'].nbytes}")
print(f"As category: {used_cars['odometer_value'].astype('category').nbytes}")

As float: 308248
As category: 125566

Kategorier kan vara besvärliga

.str-accessorn konverterar en Series till object vid datamanipulering.
Metoden .apply() returnerar en ny Series som object.
De vanliga metoderna för att lägga till, ta bort, ersätta eller ange kategorier hanterar saknade kategorier på olika sätt.
NumPy-funktioner fungerar generellt inte med kategoriska Series.

Kontrollera och konvertera

Kontrollera

used_cars["color"] = used_cars["color"].astype("category")
used_cars["color"] = used_cars["color"].str.upper()
print(used_cars["color"].dtype)

object

Konvertera

used_cars["color"] = used_cars["color"].astype("category")
print(used_cars["color"].dtype)

category

Leta efter saknade värden

Ange kategorier

used_cars["color"] = used_cars["color"].astype("category")
used_cars["color"].cat.set_categories(["black", "silver", "blue"], inplace=True)

used_cars["color"].value_counts(dropna=False)

NaN       18172
black      7705
silver     6852
blue       5802
Name: color, dtype: int64

Använda NumPy-arrayer

used_cars['number_of_photos'] = used_cars['number_of_photos'].astype("category")
used_cars['number_of_photos'].sum()  # <--- Gives an Error

TypeError: Categorical cannot perform the operation sum

used_cars['number_of_photos'].astype(int).sum()

Obs:

# .str converts the column to an array
used_cars["color"].str.contains("red")

0        False
1        False
...

Nu kör vi en övning!

Arbeta med kategoriska data i Python