Introduction to Dataflows Gen2

Data Ingestion and Semantic Models with Microsoft Fabric

Alex Kuntz

Head of Cloud Curriculum, DataCamp

Dataflows Gen2 in Microsoft Fabric

  • What Are Dataflows Gen2?

    • Tool for data transformation and ingestion

    Dataflows Gen 2

  • Key Features:

    • Supports 300 plus Complex Transformations (e.g., merging, pivoting)
    • Flexible schema options for dynamic customization
    • Staging & Fast Copy: Handling large data volumes
    • Seamless integration with Data Pipelines
Data Ingestion and Semantic Models with Microsoft Fabric

Overview of the DataFlows Gen2 Interface

DataFlows Gen2 leverages Power Query Online for an intuitive transformation experience. Dataflow Gen 2 Interface

Data Ingestion and Semantic Models with Microsoft Fabric

Dataflow Gen 2 Interface Components

  • Power Query Ribbon - Connect data sources and apply transformations Power Query Ribbon

  • Queries Pane: Manage, rename, and enable staging for data sources.

Queries

Data Ingestion and Semantic Models with Microsoft Fabric

Dataflow Gen 2 Interface Components

  • Diagram View: Visualize data sources and transformations Diagram View

  • Data Preview Pane: Preview and interact with data

    Data Preview Pane

Data Ingestion and Semantic Models with Microsoft Fabric

Dataflow Gen 2 Interface Components

  • Query Settings Pane: Track and modify applied steps

Query settings

Data Ingestion and Semantic Models with Microsoft Fabric

Supported Data Destinations in DataFlows Gen2

DataFlow Gen2 allows adding destination for each query, offering flexibility in data storage.

Supported Data Destinations:

  • Azure SQL Databases
  • Azure Data Explorer (Kusto)
  • Fabric Lakehouse
  • Fabric Warehouse
  • Fabric KQL Database

Supported Destination for Dataflows Gen2

Data Ingestion and Semantic Models with Microsoft Fabric

Integrating Dataflows Gen2 in Data Pipelines

Why Integrate?
Perform further operations on transformed data and automate workflows

dataflow pipeline activity

  • Add Dataflows Gen2 as a pipeline activity.
  • Ideal for additional operations on transformed data.
  • Automate Dataflows within the pipeline.
  • Monitor performance in real-time.

Integratining DataFlows in Data Pipeline

Data Ingestion and Semantic Models with Microsoft Fabric

Let's practice!

Data Ingestion and Semantic Models with Microsoft Fabric

Preparing Video For Download...