时间序列数据插补
在 Python 中处理缺失数据
Suraj Donthi
Deep Learning & Computer Vision Consultant
Airquality 数据集
import pandas as pd airquality = pd.read_csv('air-quality.csv', parse_dates='Date', index_col='Date')airquality.head()
Ozone Solar Wind Temp
Date
1976-05-01 41.0 190.0 7.4 67
1976-05-02 36.0 118.0 8.0 72
1976-05-03 12.0 149.0 12.6 74
1976-05-04 18.0 313.0 11.5 62
1976-05-05 NaN NaN 14.3 56
Airquality 数据集
airquality.isnull().sum()
Ozone 37
Solar 7
Wind 0
Temp 0
dtype: int64
airquality.isnull.mean() * 100
Ozone 24.183007
Solar 4.575163
Wind 0.000000
Temp 0.000000
dtype: float64
.fillna() 方法
.fillna() 的 method 可设置为
'ffill'或'pad''bfill'或'backwardfill'
前向填充（ffill）
- 用上一个观测值替换
NaN pad等同于'ffill'
airquality.fillna(method='ffill', inplace=True)
airquality['Ozone'][30:40]
Date Ozone
1976-05-31 37.0
1976-06-01 NaN
1976-06-02 NaN
1976-06-03 NaN
1976-06-04 NaN
1976-06-05 NaN
1976-06-06 NaN
1976-06-07 29.0
1976-06-08 NaN
1976-06-09 71.0
airquality.fillna(method='ffill',
inplace=True)
airquality['Ozone'][30:40]
Date Ozone
1976-05-31 37.0
1976-06-01 37.0
1976-06-02 37.0
1976-06-03 37.0
1976-06-04 37.0
1976-06-05 37.0
1976-06-06 37.0
1976-06-07 29.0
1976-06-08 29.0
1976-06-09 71.0
后向填充（bfill）
- 用下一个观测值替换
NaN backfill等同于'bfill'
df.fillna(method='bfill', inplace=True)
airquality['Ozone'][30:40]
Date Ozone
1976-05-31 37.0
1976-06-01 NaN
1976-06-02 NaN
1976-06-03 NaN
1976-06-04 NaN
1976-06-05 NaN
1976-06-06 NaN
1976-06-07 29.0
1976-06-08 NaN
1976-06-09 71.0
airquality.fillna(method='bfill',
inplace=True)
airquality['Ozone'][30:40]
Date Ozone
1976-05-31 37.0
1976-06-01 29.0
1976-06-02 29.0
1976-06-03 29.0
1976-06-04 29.0
1976-06-05 29.0
1976-06-06 29.0
1976-06-07 29.0
1976-06-08 71.0
1976-06-09 71.0
.interpolate() 方法
.interpolate()将数值序列延伸到缺失位置
.interpolate() 的 method 可设置为
'linear''quadratic''nearest'
线性插值
- 线性或等距插补缺失值
df.interpolate(method='linear', inplace=True)
airquality['Ozone'][30:40]
Date Ozone
1976-05-31 37.0
1976-06-01 NaN
1976-06-02 NaN
1976-06-03 NaN
1976-06-04 NaN
1976-06-05 NaN
1976-06-06 NaN
1976-06-07 29.0
1976-06-08 NaN
1976-06-09 71.0
airquality.interpolate(
method='linear', inplace=True)
airquality['Ozone'][30:40]
Date Ozone
1976-05-31 37.0
1976-06-01 35.9
1976-06-02 34.7
1976-06-03 33.6
1976-06-04 32.4
1976-06-05 31.3
1976-06-06 30.1
1976-06-07 29.0
1976-06-08 50.0
1976-06-09 71.0
二次插值
- 用二次函数插补缺失值
df.interpolate(method='quadratic', inplace=True)
airquality['Ozone'][30:39]
Ozone
Date
1976-05-31 37.0
1976-06-01 NaN
1976-06-02 NaN
1976-06-03 NaN
1976-06-04 NaN
1976-06-05 NaN
1976-06-06 NaN
1976-06-07 29.0
1976-06-08 NaN
airquality.interpolate(
method='quadratic', inplace=True)
airquality['Ozone'][30:39]
Ozone
Date
1976-05-31 37.0
1976-06-01 -38.4
1976-06-02 -79.4
1976-06-03 -85.9
1976-06-04 -62.4
1976-06-06 -2.8
1976-06-07 29.0
1976-06-08 62.2
最近值插补
- 用最近的观测值插补
df.interpolate(method='nearest', inplace=True)
airquality['Ozone'][30:39]
Date Ozone
1976-05-31 37.0
1976-06-01 NaN
1976-06-02 NaN
1976-06-03 NaN
1976-06-04 NaN
1976-06-05 NaN
1976-06-06 NaN
1976-06-07 29.0
1976-06-08 NaN
airquality.interpolate(
method='nearest', inplace=True)
airquality['Ozone'][30:39]
Date Ozone
1976-05-31 37.0
1976-06-01 37.0
1976-06-02 37.0
1976-06-03 37.0
1976-06-04 29.0
1976-06-05 29.0
1976-06-06 29.0
1976-06-07 29.0
1976-06-08 29.0
Vamos praticar!
在 Python 中处理缺失数据
