Summary  
This chapter covers data flow programming by constructing pipelines of data transformations composed of source, transform, and sink operations in a declarative interface, including schema projection, query-based filtering, and runtime preview.  

General domain of usage  
ETL and data integration

**Data Flows** in Azure Data Factory are visual, code-free tools that enable complex data transformations within ETL or ELT workflows directly in ADF.

Definition

For instance, imagine a scenario where you need to clean, enrich, and aggregate sales data from multiple regions. Instead of writing extensive SQL or Python scripts, you can use a Data Flow to visually map these transformations and execute them seamlessly within ADF.

## Key Components of Data Flows
- **Source Transformation**: defines where the data originates, such as Blob Storage or a SQL Database;
- **Transformations**: include tools like filtering, joining, aggregating, or deriving new columns to manipulate the data;
- **Sink Transformation**: specifies the destination for the processed data, such as another SQL Database, a data lake, or a file storage.

We will start our work with creating simple dataflow with source and sink transformations.

Download File

## How to Set Up a Source Transformation

1. Add a **new Data Flow** in the Author section of Azure Data Factory Studio;
2. Drag a **Source Transformation** from the toolbox onto the Data Flow canvas; 
3. In the **Source Transformation settings**, select a **Linked Service**, such as Azure SQL Database or Azure Blob Storage, to connect to your data source; 
4. Choose an **existing Dataset** or create a **new Dataset** that represents the data to be ingested; 
5. Configure **file format options** if connecting to Blob Storage, or provide a **SQL query** to filter or structure the incoming data for databases; 
6. **Validate the configuration** and **preview the data** to ensure the source is correctly set up.  

## Sink Transformation for Processed Data
After defining transformations, use a Sink Transformation to specify where the transformed data will be stored. For example, you might save aggregated data back to the SQL database or export it as a CSV file to Blob Storage.

Remember, every dataflow **must include at least one** source transformation and one sink transformation.

Note

Which of the following best describes an example use case for Data Flows?


Master the essentials of data engineering with Microsoft Azure in this comprehensive course. Starting with foundational concepts like cloud computing, resource management, and storage solutions, you'll progress to hands-on training in Azure Data Factory (ADF), mastering ETL/ELT workflows, and advanced data transformations. Through practical examples and real-world problem-solving, you'll gain the skills to design, implement, and optimize scalable data solutions using Azure.

Have a deep dive into the foundational concepts and tools required to begin your journey with Microsoft Azure. You'll explore cloud computing basics, learn how to organize and manage resources efficiently using Azure Resource Groups, and set up core services like Blob Storage and Azure SQL Database. Additionally, you’ll secure sensitive data with Azure Key Vault and gain an overview of Azure Data Factory (ADF) to understand its role in modern data engineering workflows.

Explore the core functionalities of Azure Data Factory (ADF), a powerful tool for creating and automating ETL/ELT workflows. In this section, you'll learn how to configure services, datasets, and pipelines while exploring key activities such as Copy, Lookup, and Script. You'll also work with advanced features like the ForEach activity for iteration and pipeline parameters for flexibility. 

This section focuses on leveraging Azure Data Factory's data flow capabilities to transform and enrich your data. You'll learn to connect various data sources, create derived columns, and merge datasets using joins. Practical examples will guide you through filtering, sorting, conditional splits, and aggregating data to generate meaningful insights. By mastering these transformations, you'll be equipped to build robust, efficient ETL workflows tailored to real-world data engineering challenges.

Apply your knowledge to real-world data engineering scenarios in this hands-on section. You'll tackle practical challenges like loading and analyzing data for different categories, identifying patterns, and assessing financial health using advanced ADF workflows. By solving these problems, you'll gain experience in designing end-to-end data solutions, working with diverse datasets, and generating actionable insights.

Understanding Data Flows in Azure Data Factory

Key Components of Data Flows

How to Set Up a Source Transformation

Sink Transformation for Processed Data