Metadata Management

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

What is metadata?

  • Data about data

Example of metadata using a book

Understanding Modern Data Architecture

Metadata types

Type of metadata Data example Book catalog example
Technical metadata Data types, relationships, column names, data sources Book's ISBN, number of pages
Business metadata Business definitions, rules, data owner Book's title, author, publisher, genre
Operational metadata Timestamps, ETL job status, data quality metrics Date of book acquisition, condition of the book
Usage metadata Who accessed the data, when, and how it was used Who checked out the book, when, and for how long
Understanding Modern Data Architecture

Where to store your metadata?

GCP Data Catalog

  • Managed metadata service
  • Integrates with GCP services
  • Can register external metadata

AWS Glue Catalog

  • Central metadata repository
  • Integrates with AWS services
  • Crawlers can catalog external data

Data catalogs

  • Azure Data Catalog
  • Apache Atlas
  • CKAN

 

  • Datahub
  • Collibra
  • ...
Understanding Modern Data Architecture

Let's practice!

Understanding Modern Data Architecture

Preparing Video For Download...