Course Content

Introduction to Data Engineering with Azure

1. Getting Started with Azure and Core Tools

Getting Started Understanding Azure Subscriptions Organizing Resources with Resource Groups Using Azure Blob Storage for Data Storage Setting Up Azure SQL Database Managing Credentials with Azure Key Vault Introduction to Azure Data Factory

2. Foundations of Azure Data Factory

Linked Services Datasets What is ADF Pipeline?Script Activity Lookup Activity Dataset Parameters ForEach Activity Execute Pipeline Activity

3. Data Flows and Transformations in ADF

Understanding Data Flows in Azure Data Factory Derived Column Transformation Combining Data with Joins Using Conditional Split and Filtering in ADF Grouping, Select, and Sort Transformations in ADF

4. Practical Problem Solving with ADF

Challenge: Loading Data to Several Tables Solving Loading Data Challenge Challenge: Conducting Transaction Analysis Solving Conducting Analysis Challenge

Using Azure Blob Storage for Data Storage

In this chapter, we will dive into Azure Blob Storage, one of the most powerful and commonly used services for storing data in the cloud.

Azure Blob Storage is a service provided by Microsoft Azure to store large amounts of unstructured data. The term "blob" stands for Binary Large Object, and it refers to the type of data stored in Azure. Blob Storage is ideal for storing data that doesn't have a predefined model or structure, such as: text files, audio or video files, images etc.

Azure Data Lake Storage Gen2 is an enhanced version of Azure Blob Storage that provides hierarchical namespace support, which allows you to organize data more effectively and makes it easier to manage large datasets.

Key Features of Azure Data Lake Storage Gen 2 (ADLS Gen2)

Scalability: ADLS Gen2 is built to handle massive amounts of data. Whether you're working with gigabytes or petabytes, you can store and manage it all within the same account;
Hierarchical namespace: ADLS Gen2 supports a hierarchical namespace that allows you to organize data in a directory structure similar to a file system. This makes it easier to manage large sets of files, apply access control policies, and optimize the performance of data operations;
Cost efficiency: Azure offers different tiers (Hot, Cool, and Archive) that allow you to store your data in the most cost-effective manner depending on how frequently you access it.

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 4