Handling missing data
Practicing Machine Learning Interview Questions in Python
Lisa Stuart
Data Scientist
Course outline
- Chapter 1: Pre-processing and Visualization
- Missing data, Outliers, Normalization
- Chapter 2: Supervised Learning
- Feature selection, Regularization, Feature engineering
- Chapter 3: Unsupervised Learning
- Cluster algorithm selection, Feature extraction, Dimension reduction
- Chapter 4: Model Selection and Evaluation
- Model generalization and evaluation, Model selection
Machine learning (ML) pipeline
Our ML pipeline
Missing data
- Impact of different techniques
- Finding missing values
- Strategies to handle
Techniques
- Omission
- Removal of rows -->
.dropna(axis=0) - Removal of columns -->
.dropna(axis=1)
- Removal of rows -->
- Imputation
- Fill with zero -->
SimpleImputer(strategy='constant', fill_value=0) - Impute mean ->
SimpleImputer(strategy='mean') - Impute median -->
SimpleImputer(strategy='median') - Impute mode -->
SimpleImputer(strategy='most_frequent') - Iterative imputation -->
IterativeImputer()
- Fill with zero -->
Why bother?
- Reduce the probability of introducing bias
- Most ML algorithms require complete data
Effects of imputation
- Depend on:
- Missing values
- Original variance
- Presence of outliers
- Size and direction of skew
- Omission --> Can remove too much
- Zero --> Bias results downward
- Mean --> Affected more by outliers
- Median --> Better in case of outliers
- Mode and iterative imputation--> Try them out
| Function | returns |
|---|---|
df.isna().sum() |
number missing |
df['feature'].mean() |
feature mean |
.shape |
row, column dimensions |
df.columns |
column names |
.fillna(0) |
fills missing with 0 |
select_dtypes(include = [np.number] ) |
numeric columns |
select_dtypes(include = ['object'] ) |
string columns |
.fit_transform(numeric_cols) |
fits and transforms |
Effects of missing values
What are the effects of missing values in a Machine Learning (ML) setting? Select the answer that is true:
- Missing values aren't a problem since most of
sklearn's algorithms can handle them. - Removing observations or features with missing values is generally a good idea.
- Missing data tends to introduce bias that leads to misleading results so they cannot be ignored.
- Filling missing values with zero will bias results upward.
Effect of missing values: answer
What are the effects of missing values in a Machine Learning (ML) setting? The correct answer is:
- Missing data tends to introduce bias that leads to misleading results so they cannot be ignored. (Filling missing values by testing which impacts the variance of a given dataset the least is the best approach.)
Effects of missing values: incorrect answers
What are the effects of missing values in a Machine Learning (ML) setting?
- Missing values aren't a problem... (Most of
sklearn's algorithms cannot handle missing values and will throw an error.) - Removing observations or features with missing values... (Unless your dataset is large and the proportion of missing values small, removing rows or columns with missing data usually results in shrinking your dataset too much to be useful in subsequent ML.)
- Filling missing values with zero will bias results upward.(It's the opposite, filling with zero will bias results downward.)
Let's practice!
Practicing Machine Learning Interview Questions in Python
