Grouping data by category in pandas
Working with Categorical Data in Python
Kasey Jones
Research Data Scientist
The basics of .groupby(): splitting data
adult = pd.read_csv("data/adult.csv")
adult1 = adult[adult["Above/Below 50k"] == " <=50K"]
adult2 = adult[adult["Above/Below 50k"] == " >50K"]
is replaced by
groupby_object = adult.groupby(by=["Above/Below 50k"])
The basics of .groupby(): apply a function
groupby_object = adult.groupby(by=["Above/Below 50k"])
Apply a function:
groupby_object.mean()
Age fnlgwt Education Num Capital Gain ...
Above/Below 50k
<=50K 36.783738 190340.86517 9.595065 148.752468 ...
>50K 44.249841 188005.00000 11.611657 4006.142456 ...
One liner:
adult.groupby(by=["Above/Below 50k"]).mean()
Specifying columns
Option 1: only runs .sum() on two columns.
adult.groupby(by=["Above/Below 50k"])['Age', 'Education Num'].sum()
Age Education Num
Above/Below 50k
<=50K 909294 237190
>50K 346963 91047
Option 2: runs .sum() on all numeric columns and then subsets.
adult.groupby(by=["Above/Below 50k"]).sum()[['Age', 'Education Num']]
Option 1 is preferred - especially when using large datasets
Groupby multiple columns
adult.groupby(by=["Above/Below 50k", "Marital Status"]).size()
Above/Below 50k Marital Status
<=50K Divorced 3980
Married-AF-spouse 13
Married-civ-spouse 8284
Married-spouse-absent 384
Never-married 10192
Separated 959
Widowed 908
>50K Divorced 463
Married-AF-spouse 10 <--- Only 10 records
Married-civ-spouse 6692
...
Practice using .groupby()
Working with Categorical Data in Python
