跨字段校验

Python 数据清洗

Adel Nehme

VP of AI Curriculum, DataCamp

动机

import pandas as pd

flights = pd.read_csv('flights.csv')
flights.head()

  flight_number  economy_class  business_class  first_class  total_passengers
0         DL140            100              60           40               200
1         BA248            130             100           70               300
2        MEA124            100              50           50               200
3        AFR939            140              70           90               300
4        TKA101            130             100           20               250

跨字段校验

使用数据集中的多个字段来检验数据一致性

  flight_number  economy_class  business_class  first_class  total_passengers
0         DL140            100       +      60      +    40        =      200
1         BA248            130       +     100      +    70        =      300
2        MEA124            100       +      50      +    50        =      200
3        AFR939            140       +      70      +    90        =      300
4        TKA101            130       +     100      +    20        =      250

sum_classes = flights[['economy_class', 'business_class', 'first_class']].sum(axis = 1)

passenger_equ = sum_classes == flights['total_passengers']

# 查找并筛除乘客总数不一致的行
inconsistent_pass = flights[~passenger_equ]
consistent_pass = flights[passenger_equ]

跨字段校验

users.head()

   user_id  Age   Birthday
0    32985   22 1998-03-02
1    94387   27 1993-12-04
2    34236   42 1978-11-24
3    12551   31 1989-01-03
4    55212   18 2002-07-02

跨字段校验

import pandas as pd
import datetime as dt

# 转为 datetime 并获取今天日期
users['Birthday'] = pd.to_datetime(users['Birthday'])

today = dt.date.today()

# 对 Birthday 每行计算年份差
age_manual = today.year - users['Birthday'].dt.year

# 查找年龄匹配的情况
age_equ = age_manual == users['Age']

# 查找并筛除年龄不一致的行
inconsistent_age = users[~age_equ]
consistent_age = users[age_equ]

发现不一致时如何处理？

Let's practice!

Python 数据清洗