Data Processing in Azure

Understanding Microsoft Azure

Kevin James

Technical Lead and Training Architect

Real-time vs Batch Processing

  • Consider processing type before choosing a service: real-time or batch.
  • Real-time: Immediate analytics
  • Batch: Scheduled or ad-hoc analytics

image of stop watch and a sand timer

Understanding Microsoft Azure

Real-time vs Batch Processing

  • Consider processing type before choosing a service
  • Real-time: Immediate analytics
  • Batch: Scheduled or ad-hoc analytics
  • Example in healthcare:
    • Real-time - hospital emergency dashboards
    • Batch - weekly-updated dashboards
  • Different infrastructure and cost implications

Healthcase analytics.jpg

Understanding Microsoft Azure

ETL Processes

Extract section from ETL.png

Understanding Microsoft Azure

ETL Processes

Extract and Transform from ETL.png

Understanding Microsoft Azure

ETL Processes

Full ETL sequence.png

Understanding Microsoft Azure

Processing tools

Processing tools image.png

Understanding Microsoft Azure

Azure Synapse Analytics

abstract azure synaps logo

  • Part of Microsoft Fabric
    • integrates big data and data warehouses
  • Unified experience for data ingestion, preparation, management, and delivery
  • Supports real-time insights and batch processing
  • Acts as a turbocharged analytics engine
Understanding Microsoft Azure

Azure Stream Analytics

abstract azure stream analytics logo

  • Enables real-time data access
  • Sets up real-time analytics with straightforward query definition
  • Handles data streaming from diverse inputs like blob storage
  • Essential for immediate insights:
    • fraud detection in a bank
    • dynamic pricing on the stock market
Understanding Microsoft Azure

Azure Databricks

  • Microsoft-Databricks collaboration
    • Analytics platform optimized for Azure
  • Unified environment for data engineering, analytics, and machine learning
  • Collaborative workspace for data scientists and engineers
  • Built-in Data Lake support
  • Real-time and batch

abstract azure databricks logo

Understanding Microsoft Azure

Azure Data Factory

abstract data factory logo thats shaped like a blue factory

  • Cloud-based integration service
  • Creates, schedules, orchestrates data workflows
  • Streamlines ETL processes
  • Handles diverse data sources and formats
  • Automates workflows for flexibility
Understanding Microsoft Azure

Azure HDInsight

abstract azure hd insight logo

  • Managed service for fast, customizable data processing
  • Runs on popular open-source platforms:
    • Hadoop
    • Spark
    • Kafka
  • Easily scales resources based on demand
  • Seamlessly connects with Azure storage solutions
Understanding Microsoft Azure

Let's practice!

Understanding Microsoft Azure

Preparing Video For Download...