Data range constraints

Cleaning Data in Python

Adel Nehme

Content Developer @ DataCamp

Motivation

movies.head()

     movie_name     avg_rating
0    The Godfather           5
1    Frozen 2                3
2    Shrek                   4
...

Motivation

import matplotlib.pyplot as plt
plt.hist(movies['avg_rating'])
plt.title('Average rating of movies (1-5)')

heart_rate_normal

Motivation

Can future sign-ups exist?

# Import date time
import datetime as dt
today_date = dt.date.today()
user_signups[user_signups['subscription_date'] > dt.date.today()]

   subscription_date  user_name         ...             Country
0         01/05/2021      Marah         ...             Nauru
1         09/08/2020     Joshua         ...             Austria
2         04/01/2020      Heidi         ...             Guinea
3         11/10/2020       Rina         ...             Turkmenistan
4         11/07/2020  Christine         ...             Marshall Islands
5         07/07/2020     Ayanna         ...             Gabon

How to deal with out of range data?

Dropping data
Setting custom minimums and maximums
Treat as missing and impute
Setting custom value depending on business assumptions

Movie example

import pandas as pd
# Output Movies with rating > 5
movies[movies['avg_rating'] > 5]

         movie_name  avg_rating
23  A Beautiful Mind           6
65   La Vita e Bella           6
77            Amelie           6

# Drop values using filtering
movies = movies[movies['avg_rating'] <= 5]

# Drop values using .drop()
movies.drop(movies[movies['avg_rating'] > 5].index, inplace = True)

# Assert results
assert movies['avg_rating'].max() <= 5

Movie example

# Convert avg_rating > 5 to 5
movies.loc[movies['avg_rating'] > 5, 'avg_rating'] = 5

# Assert statement
assert movies['avg_rating'].max() <= 5

Remember, no output means it passed

Date range example

import datetime as dt
import pandas as pd
# Output data types
user_signups.dtypes

subscription_date    object
user_name            object
Country              object
dtype: object

# Convert to date
user_signups['subscription_date'] = pd.to_datetime(user_signups['subscription_date']).dt.date

Date range example

today_date = dt.date.today()

Drop the data

# Drop values using filtering
user_signups = user_signups[user_signups['subscription_date'] < today_date]

# Drop values using .drop()
user_signups.drop(user_signups[user_signups['subscription_date'] > today_date].index, inplace = True)

Hardcode dates with upper limit

# Drop values using filtering
user_signups.loc[user_signups['subscription_date'] > today_date, 'subscription_date'] = today_date
# Assert is true
assert user_signups.subscription_date.max().date() <= today_date

Let's practice!

Cleaning Data in Python