Data Pipelines

DevOps Concepts

Cem Sakarya

DevOps Risk Advisor

Data pipelines

Microservices communicate to each other via APIs. Also they interact with independent databases through data pipelines. Data pipelines are highlighted.

ETL

Extract; Transform; Load

¹ https://www.istockphoto.com/nl/vector/data-extract-transform-load-gm1361894912-434102842?phrase=ETL

Batch processing

Batch Processing Data Pipeline taking data from a microservice, does operations on it and load it to an independent database.

Move offline data in batches
Batch Processing: A large amount of data

Multiple users connect to the backend at the same time.

User connections are ingested into the backend by the ingestion API.

User connections are routed to different microservices via a streaming data pipeline.

Recap

Data Engineering is an integral part of Infrastructure Engineering
Data pipelines are robust tools important for handling data
Batch processing works on regular schedules, stream processing works continuously
Batch processing is useful for handling the accumulated data
Stream processing handles the real-time data

Let's practice!

DevOps Concepts