Is the data missing at random?

Dealing with Missing Data in Python

Suraj Donthi

Deep Learning & Computer Vision Consultant

Possible reasons for missing data

Note $-$ (variable $\rightarrow$ data field or column in a DataFrame)

  • Values simply missing at random instances or intervals in a variable
  • Values missing due to another variable
  • Values missing due to the missingness of the same or another variable
Dealing with Missing Data in Python

Types of missingness

  1. Missing Completely at Random (MCAR)
  2. Missing at Random (MAR)
  3. Missing Not at Random (MNAR)
Dealing with Missing Data in Python

Missing Completely at Random(MCAR)

Definition:

"Missingness has no relationship between any values, observed or missing"

Example for Missing Completely at Random

Dealing with Missing Data in Python

MCAR - An example

msno.matrix(diabetes)

Missing Completely at Random for diabetes dataset

Dealing with Missing Data in Python

Missing at Random(MAR)

Definition:

"There is a systematic relationship between missingness and other observed data, but not the missing data"

Example for Missing at Random

Dealing with Missing Data in Python

MAR - An example

msno.matrix(diabetes)

Missing at Random for diabetes dataset

Dealing with Missing Data in Python

Missing not at Random(MNAR)

Definition:

"There is a relationship between missingness and its values, missing or non-missing"

Example for Missing Not at Random

Dealing with Missing Data in Python

MNAR - An example

  • Missingness pattern of the diabetes sorted by Serum_Insulin
sorted = diabetes.sort_values('Serum_Insulin')
msno.matrix(sorted)

Missing Not at Random for diabetes dataset

Dealing with Missing Data in Python

Summary

  • Possible reasons for missingness
    • Missing Completely at Random (MCAR),
    • Missing at Random (MAR) or
    • Missing Not at Random (MNAR)
  • Detecting missingness pattern by sorting the variables
  • Mapping missingness to MCAR, MAR & MNAR
Dealing with Missing Data in Python

Let's practice!

Dealing with Missing Data in Python

Preparing Video For Download...