Data Mesh and Data Fabric

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

Data Mesh

  • Architectural approach to decentralize data ownership and management
  • Ownership and management:
    • Full responsibility for the data
    • Ensure quality, security and privacy
Understanding Modern Data Architecture

Data Mesh architecture

  • Owners for each dataset
  • Expose data as products
  • Sharing a common infrastructure
  • Microservices
  • Mindset shift!

Data mesh architecture from 30,000 foot view

1 https://martinfowler.com/articles/data-monolith-to-mesh.html
Understanding Modern Data Architecture

How to process data then?

  • Organizational and cultural aspects of data management
  • Domain data products are first-class citizens
  • Tools are second-class citizens
  • Data lakes and warehouses are just nodes, details within a domain
Understanding Modern Data Architecture

Data Mesh benefits and challenges

  • Benefits:
    • Data democratization
    • Decentralization
    • Clear ownership
    • Scalability and agility
    • Faster Time to Market
    • Flexibility in Technology Stack
    • ...
  • Challenges:
    • Cultural Shift
    • Data Governance
    • Collaboration and Coordination
    • Technical Complexity
Understanding Modern Data Architecture

Data Fabric

  • Unified and integrated view
  • Democratization of data
  • Metadata
    • Knowledge graph
    • AI/ML algorithms
    • Active metadata
  • Intelligent integration
  • Better data management and protection
1 https://www.gartner.com/smarterwithgartner/data-fabric-architecture-is-key-to-modernizing-data-management-and-integration 2 https://www.ibm.com/topics/data-fabric
Understanding Modern Data Architecture

A Data Fabric should support...

  • ETL
  • Streaming
  • Replication
  • Messaging
  • Data virtualization
  • Data microservices
  • ...
Understanding Modern Data Architecture

Let's practice!

Understanding Modern Data Architecture

Preparing Video For Download...