数据类型约束

Python 数据清洗

Adel Nehme

VP of AI Curriculum, DataCamp

课程概览

dirty_data

Python 数据清洗

课程概览

side effects

Python 数据清洗

课程概览

clean_data

Python 数据清洗

课程概览

clean_data

第1章:常见数据问题

Python 数据清洗

为什么需要清洗数据?

ds_workflow

Python 数据清洗

为什么需要清洗数据?

ds_workflow

Python 数据清洗

为什么需要清洗数据?

                                                                                   垃圾进,垃圾出

Python 数据清洗

数据类型约束

数据类型 示例
文本 名、姓、地址 ...
整数 订阅数、销量 ...
小数 温度、美元汇率 ...
二元 是否已婚、新客户、是/否 ...
日期 下单日期、发货日期 ...
类别 婚姻状况、性别 ...
Python 数据类型
str
int
float
bool
datetime
category
Python 数据清洗

字符串转整数

# Import CSV file and output header
sales = pd.read_csv('sales.csv')
sales.head(2)
   SalesOrderID    Revenue    Quantity
0         43659     23153$          12
1         43660      1457$           2
# Get data types of columns
sales.dtypes
SalesOrderID    int64
Revenue         object
Quantity        int64
dtype: object
Python 数据清洗

字符串转整数

# Get DataFrame information
sales.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31465 entries, 0 to 31464
Data columns (total 3 columns):
SalesOrderID     31465 non-null int64
Revenue          31465 non-null object
Quantity         31465 non-null int64
dtypes: int64(2), object(1)
memory usage: 737.5+ KB
Python 数据清洗

字符串转整数

# Print sum of all Revenue column
sales['Revenue'].sum()
'23153$1457$36865$32474$472$27510$16158$5694$6876$40487$807$6893$9153$6895$4216..
# Remove $ from Revenue column
sales['Revenue'] = sales['Revenue'].str.strip('$')
sales['Revenue'] = sales['Revenue'].astype('int')
# Verify that Revenue is now an integer
assert sales['Revenue'].dtype == 'int'

DNT_CURLLY_TAG_3

Python 数据清洗

assert 语句

# This will pass
assert 1+1 == 2
# This will not pass
assert 1+1 == 3
AssertionError                            Traceback (most recent call last)
         assert 1+1 == 3
AssertionError:
Python 数据清洗

数值还是类别?

...   marriage_status    ...
...                 3    ...
...                 1    ...
...                 2    ...

0 = 从未结婚       1 = 已婚       2 = 分居       3 = 离婚

df['marriage_status'].describe()
       marriage_status
...
mean              1.4
std               0.20
min               0.00
50%               1.8 ...
Python 数据清洗

数值还是类别?

# Convert to categorical
df["marriage_status"] = df["marriage_status"].astype('category')

df.describe()
        marriage_status
count                 241
unique                4
top                   1
freq                  120
Python 数据清洗

开始练习吧!

Python 数据清洗

Preparing Video For Download...