Introduction to Fabric Data Pipelines

Data Ingestion and Semantic Models with Microsoft Fabric

Alex Kuntz

Head of Cloud Curriculum, DataCamp

Data Pipelines in Microsoft Fabric

Two Primary High-Level Features in Data Factory:

  1. Data Pipelines - Orchestrates data movement. Data Pipelines
  2. Dataflows - supports UI-based 300 + Data transformations. Dataflows

Data Pipelines in Microsoft Fabric:

  • Automates ETL process with minimal or no code.
  • Supports seamless integration of diverse data sources
  • Rich set of activities for data ingestion and transformation
  • Run pipelines manually or schedule them with triggers
1 https://learn.microsoft.com/en-us/fabric/data-factory/data-factory-overview
Data Ingestion and Semantic Models with Microsoft Fabric

Activities in Data Pipelines

Activities are tasks within Pipelines, orchestrating data processing and workflow automation. Car assembly line illustrating activities in a data pipeline

1 https://learn.microsoft.com/en-us/fabric/data-factory/activity-overview
Data Ingestion and Semantic Models with Microsoft Fabric

Types of Activities

  1. Move & Transform:
    • Handles data transfer and transformation tasks
    • (e.g., Copy data Copy data).
  2. Metadata & Validation:
    • Manages data quality checks and metadata retrieval.
    • (e.g., LookupLookup).
  3. Control Flow:
    • Controls task sequence based on conditions and loops.
    • (e.g., If condition If condition, ForEach If condition).
1 https://learn.microsoft.com/en-us/fabric/data-factory/activity-overview
Data Ingestion and Semantic Models with Microsoft Fabric

Types of Activities

  1. Orchestrate:

    • Synchronize multiple processes
    • (e.g., Invoke Pipeline Invoke Pipeline).

  2. Notifications:

    • Sends alerts and updates through email or messaging tools
    • (e.g., 365 Outlook365 Outlook, TeamsTeams).

  3. Transform:

    • Execute data manipulations as per the business logic.
    • (e.g., NotebookNotebook, Stored ProcedureStored Procedure).
Data Ingestion and Semantic Models with Microsoft Fabric

Pipeline Parameters and Variables

Parameters and Variables help control and manage pipeline behavior dynamically.

Parameters:

  • Set at Runtime: Adjust pipeline behavior using external inputs.
  • Global Scope: Influence the entire pipeline execution.

Variables:

  • Dynamic Tracking: Change values during pipeline execution.
  • Local Scope: Manage data within specific pipeline activities.
1 https://learn.microsoft.com/en-us/fabric/data-factory/parameters 2 https://learn.microsoft.com/en-us/fabric/data-factory/set-variable-activity
Data Ingestion and Semantic Models with Microsoft Fabric

Pipeline runs

A pipeline run executes the activities in your pipeline to completion.

  • On-Demand: Start pipelines directly from the Fabric UI. On-Demand Run
  • Scheduled: Starts the run at a specific frequency. Scheduled Run

Monitoring:

  • Track and review each pipeline run using its unique Run ID in the Monitor Tab.

Validation:

  • Ensure your pipeline configuration is correct by using the Validate option before execution.
1 https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-runs
Data Ingestion and Semantic Models with Microsoft Fabric

Let's practice!

Data Ingestion and Semantic Models with Microsoft Fabric

Preparing Video For Download...