Imputatie van tijdreeksen

Omgaan met ontbrekende gegevens in Python

Suraj Donthi

Deep Learning & Computer Vision Consultant

Airquality-dataset

import pandas as pd
airquality = pd.read_csv('air-quality.csv', parse_dates='Date', 
                                index_col='Date')

airquality.head()

             Ozone    Solar    Wind    Temp
Date                
1976-05-01    41.0    190.0     7.4    67
1976-05-02    36.0    118.0     8.0    72
1976-05-03    12.0    149.0    12.6    74
1976-05-04    18.0    313.0    11.5    62
1976-05-05     NaN      NaN    14.3    56

Airquality-dataset

airquality.isnull().sum()

Ozone    37
Solar     7
Wind      0
Temp      0
dtype: int64

airquality.isnull.mean() * 100

Ozone    24.183007
Solar     4.575163
Wind      0.000000
Temp      0.000000
dtype: float64

De methode .fillna()

De parameter method in .fillna() kan zijn

'ffill' of 'pad'
'bfill' of 'backwardfill'

Methode ffill

Vervang NaNs door laatst bekende waarde
pad is hetzelfde als 'ffill'

airquality.fillna(method='ffill', inplace=True)



airquality['Ozone'][30:40]

Date         Ozone        
1976-05-31    37.0
1976-06-01     NaN
1976-06-02     NaN
1976-06-03     NaN
1976-06-04     NaN
1976-06-05     NaN
1976-06-06     NaN
1976-06-07    29.0
1976-06-08     NaN
1976-06-09    71.0

airquality.fillna(method='ffill', 
                         inplace=True)
airquality['Ozone'][30:40]

Date         Ozone        
1976-05-31    37.0
1976-06-01    37.0
1976-06-02    37.0
1976-06-03    37.0
1976-06-04    37.0
1976-06-05    37.0
1976-06-06    37.0
1976-06-07    29.0
1976-06-08    29.0
1976-06-09    71.0

Methode bfill

Vervang NaNs door eerstvolgende waarde
backfill is hetzelfde als 'bfill'

df.fillna(method='bfill', inplace=True)



airquality['Ozone'][30:40]

Date         Ozone        
1976-05-31    37.0
1976-06-01     NaN
1976-06-02     NaN
1976-06-03     NaN
1976-06-04     NaN
1976-06-05     NaN
1976-06-06     NaN
1976-06-07    29.0
1976-06-08     NaN
1976-06-09    71.0

airquality.fillna(method='bfill', 
                         inplace=True)
airquality['Ozone'][30:40]

Date         Ozone        
1976-05-31    37.0
1976-06-01    29.0
1976-06-02    29.0
1976-06-03    29.0
1976-06-04    29.0
1976-06-05    29.0
1976-06-06    29.0
1976-06-07    29.0
1976-06-08    71.0
1976-06-09    71.0

De methode .interpolate()

De methode .interpolate() vult ontbrekende waarden in de reeks aan

De parameter method in .interpolate() kan zijn

'linear'
'quadratic'
'nearest'

Lineaire interpolatie

Lineair of met equidistante waarden imputeren

df.interpolate(method='linear', inplace=True)

snippet lineaire interpolatie



airquality['Ozone'][30:40]

Date         Ozone        
1976-05-31    37.0
1976-06-01     NaN
1976-06-02     NaN
1976-06-03     NaN
1976-06-04     NaN
1976-06-05     NaN
1976-06-06     NaN
1976-06-07    29.0
1976-06-08     NaN
1976-06-09    71.0

airquality.interpolate(
          method='linear', inplace=True)
airquality['Ozone'][30:40]

Date         Ozone        
1976-05-31    37.0
1976-06-01    35.9
1976-06-02    34.7
1976-06-03    33.6
1976-06-04    32.4
1976-06-05    31.3
1976-06-06    30.1
1976-06-07    29.0
1976-06-08    50.0
1976-06-09    71.0

Kwadratische interpolatie

Imputeer kwadratisch

df.interpolate(method='quadratic', inplace=True)

snippet kwadratische interpolatie



airquality['Ozone'][30:39]

             Ozone
Date                
1976-05-31    37.0
1976-06-01     NaN
1976-06-02     NaN
1976-06-03     NaN
1976-06-04     NaN
1976-06-05     NaN
1976-06-06     NaN
1976-06-07    29.0
1976-06-08     NaN

airquality.interpolate(
  method='quadratic', inplace=True)
airquality['Ozone'][30:39]

             Ozone
Date                
1976-05-31    37.0
1976-06-01   -38.4
1976-06-02   -79.4
1976-06-03   -85.9
1976-06-04   -62.4
1976-06-06    -2.8
1976-06-07    29.0
1976-06-08    62.2

Imputatie met dichtstbijzijnde waarde

Imputeer met de dichtstbijzijnde waarneming

df.interpolate(method='nearest', inplace=True)

snippet dichtstbijzijnde interpolatie



airquality['Ozone'][30:39]

Date         Ozone        
1976-05-31    37.0
1976-06-01     NaN
1976-06-02     NaN
1976-06-03     NaN
1976-06-04     NaN
1976-06-05     NaN
1976-06-06     NaN
1976-06-07    29.0
1976-06-08     NaN

airquality.interpolate(
  method='nearest', inplace=True)
airquality['Ozone'][30:39]

Date         Ozone        
1976-05-31    37.0
1976-06-01    37.0
1976-06-02    37.0
1976-06-03    37.0
1976-06-04    29.0
1976-06-05    29.0
1976-06-06    29.0
1976-06-07    29.0
1976-06-08    29.0

Laten we oefenen!

Omgaan met ontbrekende gegevens in Python