Designing a modern data architecture

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

The business case

Medical laboratory

  • Organizer robot
  • Generate CSV
    • Up to 4 CSV every hour
  • Databases in plain files

Business case flow

  • Platform for patients to track results
  • Enrich patients results with their investigation
Understanding Modern Data Architecture

Where to start?

Questions!

Refine the requirements!

  • How large are the files?
  • How many robots?
  • How frequently are those files generated?
  • How many files do they have?
  • How data will be processed? Machine learning? Queries?
  • How data will be exposed?
  • What regulations/constraints do we have?
Understanding Modern Data Architecture

The assumptions

  • 100 machines
  • Each CSV file is around 100MB
  • Plain files as database
  • Tens of gigabytes for each plain file
  • Model exposed through API
    • Requires all previous result or summary
    • Summary needs to be updated constantly
  • Mobile app
  • Ignore regulations
Understanding Modern Data Architecture

The solution

Proposed solution

Understanding Modern Data Architecture

Let's practice!

Understanding Modern Data Architecture

Preparing Video For Download...