Datasets
Swipe to show menu
When working with data in Azure Data Factory, datasets play a crucial role. They define the data structures or schema used in pipelines, essentially serving as a pointer to the data you want to move or transform. Datasets work in tandem with Linked Services to help ADF access, process, and manage data efficiently.
Formally, a dataset in ADF represents a specific dataset or table-like structure within a data source or destination. For instance, a dataset could be a folder or file in Azure Blob Storage, or it could also represent a table in Azure SQL Database.
While Linked Services define where the data resides, datasets specify what part of the data to work with.
Why Are Datasets Important?
- Data interaction: datasets provide the schema and parameters that help ADF understand and interact with your data;
- Reusability: you can define datasets once and reuse them across multiple activities in pipelines;
- Flexibility: datasets allow dynamic configurations using parameters, enabling pipelines to handle variable data inputs or outputs.
In this chapter, we created datasets that represent our data in Azure Storage and Azure Database without any parameterizations.
Thanks for your feedback!
Ask AI
Ask AI
Ask anything or try one of the suggested questions to begin our chat
Datasets
When working with data in Azure Data Factory, datasets play a crucial role. They define the data structures or schema used in pipelines, essentially serving as a pointer to the data you want to move or transform. Datasets work in tandem with Linked Services to help ADF access, process, and manage data efficiently.
Formally, a dataset in ADF represents a specific dataset or table-like structure within a data source or destination. For instance, a dataset could be a folder or file in Azure Blob Storage, or it could also represent a table in Azure SQL Database.
While Linked Services define where the data resides, datasets specify what part of the data to work with.
Why Are Datasets Important?
- Data interaction: datasets provide the schema and parameters that help ADF understand and interact with your data;
- Reusability: you can define datasets once and reuse them across multiple activities in pipelines;
- Flexibility: datasets allow dynamic configurations using parameters, enabling pipelines to handle variable data inputs or outputs.
In this chapter, we created datasets that represent our data in Azure Storage and Azure Database without any parameterizations.
Thanks for your feedback!
