分类变量

Python 数据清洗

Adel Nehme

Content Developer @DataCamp

可能出现哪些错误？

I) 取值不一致

不一致字段： 'married'、'Maried'、'UNMARRIED'、'not married' 等
尾随空格： 'married '、' married ' 等

II) 将过多类别合并为更少

创建新分组： 从连续的家庭收入数据生成 0-20K、20-40K 等类别
映射到新分组： 将家庭收入类别映射为两组 'rich'、'poor'

III) 确保数据类型为 category（见第 1 章）

取值一致性

大小写： 'married'、'Married'、'UNMARRIED'、'unmarried' 等

# 获取婚姻状态列
marriage_status = demographics['marriage_status']
marriage_status.value_counts()

unmarried    352
married      268
MARRIED      204
UNMARRIED    176
dtype: int64

取值一致性

# 对 DataFrame 计数
marriage_status.groupby('marriage_status').count()

                 household_income  gender
marriage_status                          
MARRIED                       204     204
UNMARRIED                     176     176
married                       268     268
unmarried                     352     352

取值一致性

# 转为大写

marriage_status['marriage_status'] = marriage_status['marriage_status'].str.upper()
marriage_status['marriage_status'].value_counts()

UNMARRIED    528
MARRIED      472

# 转为小写

marriage_status['marriage_status'] = marriage_status['marriage_status'].str.lower()
marriage_status['marriage_status'].value_counts()

unmarried    528
married      472

取值一致性

尾随空格： 'married '、'married'、'unmarried'、' unmarried' 等

# 获取婚姻状态列
marriage_status = demographics['marriage_status']
marriage_status.value_counts()

 unmarried   352
unmarried    268
married      204
married      176
dtype: int64

取值一致性

# 去除所有空格
demographics = demographics['marriage_status'].str.strip()
demographics['marriage_status'].value_counts()

unmarried    528
married      472

将数据折叠为类别

从数据创建分组：从 income 列生成 income_group 列。

# 使用 qcut()
import pandas as pd
group_names = ['0-200K', '200K-500K', '500K+']
demographics['income_group'] = pd.qcut(demographics['household_income'], q = 3, 
                                       labels = group_names)
# 打印 income_group 列
demographics[['income_group', 'household_income']]

     category  household_income
0   200K-500K  189243
1       500K+  778533
..

将数据折叠为类别

从数据创建分组：从 income 列生成 income_group 列。

# 使用 cut()——创建区间与名称
ranges = [0,200000,500000,np.inf]
group_names = ['0-200K', '200K-500K', '500K+']
# 创建收入分组列
demographics['income_group'] = pd.cut(demographics['household_income'], bins=ranges, 
                                      labels=group_names)
demographics[['income_group', 'household_income']]

     category  Income
0      0-200K  189243
1       500K+  778533

将数据折叠为类别

映射合并类别： 将分类列的多值合并为更少的类别。

operating_system 列为：'Microsoft'、'MacOS'、'IOS'、'Android'、'Linux'

operating_system 列目标：'DesktopOS'、'MobileOS'

# 创建映射字典并替换
mapping = {'Microsoft':'DesktopOS', 'MacOS':'DesktopOS', 'Linux':'DesktopOS',
           'IOS':'MobileOS', 'Android':'MobileOS'}
devices['operating_system'] = devices['operating_system'].replace(mapping)
devices['operating_system'].unique()

array(['DesktopOS', 'MobileOS'], dtype=object)

Passons à la pratique !

Python 数据清洗