Data Processing in Azure

Understanding Microsoft Azure

Kevin James

Technical Lead and Training Architect

Real-time vs Batch Processing

• Consider processing type before choosing a service: real-time or batch.
• Real-time: Immediate analytics
• Batch: Scheduled or ad-hoc analytics

Real-time vs Batch Processing

• Consider processing type before choosing a service
• Real-time: Immediate analytics
• Batch: Scheduled or ad-hoc analytics
• Example in healthcare:
  • Real-time - hospital emergency dashboards
  • Batch - weekly-updated dashboards
• Different infrastructure and cost implications

ETL Processes

ETL Processes

ETL Processes

Processing tools

Azure Synapse Analytics

• Part of Microsoft Fabric
  • integrates big data and data warehouses
• Unified experience for data ingestion, preparation, management, and delivery
• Supports real-time insights and batch processing
• Acts as a turbocharged analytics engine

Azure Stream Analytics

• Enables real-time data access
• Sets up real-time analytics with straightforward query definition
• Handles data streaming from diverse inputs like blob storage
• Essential for immediate insights:
  • fraud detection in a bank
  • dynamic pricing on the stock market

Azure Databricks

• Microsoft-Databricks collaboration
  • Analytics platform optimized for Azure
• Unified environment for data engineering, analytics, and machine learning
• Collaborative workspace for data scientists and engineers
• Built-in Data Lake support
• Real-time and batch

Azure Data Factory

• Cloud-based integration service
• Creates, schedules, orchestrates data workflows
• Streamlines ETL processes
• Handles diverse data sources and formats
• Automates workflows for flexibility

Azure HDInsight

• Managed service for fast, customizable data processing
• Runs on popular open-source platforms:
  • Hadoop
  • Spark
  • Kafka
• Easily scales resources based on demand
• Seamlessly connects with Azure storage solutions

Let's practice!

Understanding Microsoft Azure