Data quality terms and concepts

Nhập môn Chất lượng Dữ liệu

Chrissy Bloom

Head of Enterprise Data Strategy & Governance

Defining data quality

  • Data Quality: a measurement of the degree to which data is fit for purpose
  • Good data quality = trust in data
    • Better business decisions
    • Better equipped business processes
  • Often good data quality is assumed
  • Data quality needs to be measured and monitored to ensure that data is fit for use.

data table

Nhập môn Chất lượng Dữ liệu

Defining data quality dimensions

Data Quality Dimension: a measurement of a specific attribute of a data's quality

  • Use data quality dimensions to quantify how fit for purpose data is.
    • Completeness
    • Validity
    • Uniqueness
    • Consistency
    • Timeliness
    • Accuracy

cube

Nhập môn Chất lượng Dữ liệu

Completeness as a data quality dimension

square grid

Completeness:

  • Dataset level: measures the degree to which all expected records in a dataset are present.
  • Data element level: measures the degree to which all records have data populated when expected.
  • Business issues due to incomplete data:
    • Numbers may be skewed
    • Customers may be affected
Nhập môn Chất lượng Dữ liệu

Completeness example

data table

All records must have a value populated in the CustomerName field.

Nhập môn Chất lượng Dữ liệu

Validity as a data quality dimension

Validity: measures the degree to which the values in a data element are valid

  • Requires business context
  • Define list or criteria for valid values
  • Numeric measurement of validity = count of valid/total count

square grid

Nhập môn Chất lượng Dữ liệu

Validity example

data table

  • CustomerBirthDate value must be a date in the past.
  • CustomerAccountType value must be either Loan or Deposit.
  • LatestAccountOpenDate value must be a date in the past.
Nhập môn Chất lượng Dữ liệu

Uniqueness as a data quality dimension

striped square

Uniqueness: measures the degree to which the records in a dataset are not duplicated

  • Requires business context to define criteria for determining unique records
  • May need to look for duplicates in one or multiple columns to identify errors
Nhập môn Chất lượng Dữ liệu

Uniqueness example

data table

All records must have a unique CustomerID and CustomerName.

Nhập môn Chất lượng Dữ liệu

Let's practice!

Nhập môn Chất lượng Dữ liệu

Preparing Video For Download...