Combining datasources for further analysis

Analyzing IoT Data in Python

Matthias Voppichler

IT Developer

Combining data sources

print(temp.head())

                     value
timestamp                 
2018-10-03 08:00:00   16.3
2018-10-03 09:00:00   17.7
2018-10-03 10:00:00   20.2
2018-10-03 11:00:00   20.9
2018-10-03 12:00:00   21.8

print(sun.head())

                      value
timestamp                  
2018-10-03 08:00:00  1798.7
2018-10-03 08:30:00  1799.9
2018-10-03 09:00:00  1798.1
2018-10-03 09:30:00  1797.7
2018-10-03 10:00:00  1798.0

Naming columns

temp.columns = ["temperature"]
sun.columns = ["sunshine"]

print(temp.head(2))
print(sun.head(2))

                     temperature
timestamp                       
2018-10-03 08:00:00         16.3
2018-10-03 09:00:00         17.7
                     sunshine
timestamp                    
2018-10-03 08:00:00    1798.7
2018-10-03 08:30:00    1799.9

Concat

environ = pd.concat([temp, sun], axis=1)

print(environ.head())

                     temperature  sunshine
timestamp                                 
2018-10-03 08:00:00         16.3    1798.7
2018-10-03 08:30:00          NaN    1799.9
2018-10-03 09:00:00         17.7    1798.1
2018-10-03 09:30:00          NaN    1797.7
2018-10-03 10:00:00         20.2    1798.0

Resample

agg_dict = {"temperature": "max", "sunshine": "sum"}

env1h = environ.resample("1h").agg(agg_dict)
print(env1h.head())

                     temperature  sunshine
timestamp                                 
2018-10-03 08:00:00         16.3    3598.6
2018-10-03 09:00:00         17.7    3595.8
2018-10-03 10:00:00         20.2    3596.2
2018-10-03 11:00:00         20.9    3594.1
2018-10-03 12:00:00         21.8    3599.9

Fillna

env30min = environ.fillna(method="ffill")
print(env30min.head())

                     temperature  sunshine
timestamp                                 
2018-10-03 08:00:00         16.3    1798.7
2018-10-03 08:30:00         16.3    1799.9
2018-10-03 09:00:00         17.7    1798.1
2018-10-03 09:30:00         17.7    1797.7
2018-10-03 10:00:00         20.2    1798.0

Let's practice!

Analyzing IoT Data in Python