Get hands on and build a modern data lake with Azure Data Factory, Data Lake Storage, Azure SQL and more!
What you'll learn
Overview and Background of Azure Big Data Analytics Solutions
Azure Data Factory
Azure Storage Accounts
Azure SQL Databases and Upsert Operations
Integrating Data From Various File Formats
Authentication and Access: Service Principals, Managed Identities and Azure Key Vault
Data Pipelines, Copy Activity, Data Flows, Control Flow and Transformation Activities
Implementing a Modern Data Lake
Control Flow, Parameters and Variables
Real World Application of Azure Data Factory
Requirements
Microsoft Azure Subscription
Basic SQL
Description
Hello students!I am pleased to present this course on one of the most in demand data engineering tools around... Azure Data Factory!As the demand for cloud-based data integration services continues to skyrocket, there is a huge demand for professionals with knowledge of services like Azure Data Factory. By learning Azure Data Factory, users can enhance their skills and increase their job prospects in the field of data engineering and analytics.In this course you will primarily be using Azure Data Factory on Microsoft Azure in addition to other services such as Azure Blob Storage, Azure Data Lake Storage Gen 2 and Azure SQL Database.The course is packed with lectures, code-along videos and a dedicated course project. As an added benefit you will also have lifetime access to all of the lectures…This course will cover the following topics:Azure Storage Solutions such as Azure Blob Storage and Azure Data Lake Gen2 StorageThe basics of Azure Data Factory including the core components such as Linked Services, Datasets, Activities, Data Flows, Pipelines and Integration RuntimesIntegrating data from various file formats such as CSV, JSON and ParquetThe Copy Activity in Azure Data FactoryData Flows and Control Flow and Transformation Activities in Azure Data FactoryOrchestrating Data Integration WorkflowsHow to create Schedules and Triggers to execute your pipelinesHow to use Parameters and Variables with your Linked Services, Datasets and PipelinesHow to use Azure Data Factory with SQL DatabasesAuthentication and Access including Managed Identities, Service Principals and Azure Key Vault
Overview
Section 1: Course Overview and Set Up
Lecture 1 Course Overview
Lecture 2 Azure Big Data Analytics
Lecture 3 Azure Account Set Up
Lecture 4 Azure UI Overview and Creating Your Subscription
Lecture 5 Services and Regions
Lecture 6 Naming Conventions and Creating the Course Resource Group
Lecture 7 Cost Management and Pricing
Lecture 8 Links and Resources
Section 2: Azure Storage
Lecture 9 Azure Storage Solutions Overview
Lecture 10 Classification of Data
Lecture 11 ADF Supported File Formats
Lecture 12 Customer Orders Dataset Download and Overview
Lecture 13 Blueprint for the Data Lake
Lecture 14 Storage Account Creation
Lecture 15 Container Creation and File Upload
Lecture 16 Links and Resources
Section 3: ADF - Overview and Set Up
Lecture 17 Azure Data Factory Overview
Lecture 18 ADF Resource Creation
Lecture 19 ADF User Interface and Components Overview
Lecture 20 Links and Resources
Section 4: Data Movement
Lecture 21 Copy Activity Overview
Lecture 22 Section Objective
Lecture 23 Copy Activity - Order Items Part 1 (Linked Services and Source/Sink Datasets)
Lecture 24 Copy Activity - Order Items Part 2 (Copy Activity and Pipeline)
Lecture 25 Copy Activity - Customers JSON
Lecture 26 Copy Activity - Orders Parquet
Lecture 27 Copy Activity - Stores and Products JSON
Lecture 28 Organising Data Factory Objects in Folders
Lecture 29 Chaining Activities
Lecture 30 Working with Multiple Files as a Source and Copy Behaviour
Lecture 31 Links and Resources
Section 5: Data Transformation
Lecture 32 Data Flow Overview
Lecture 33 Section Objective
Lecture 34 Data Flow Demo and User Interface
Lecture 35 Data Flow Debug (Pricing)
Lecture 36 Data Flow Debug (Row Limit)
Lecture 37 Source and Sink Partitioning
Lecture 38 Select Transformation
Lecture 39 Cast Transformation
Lecture 40 Import Projection
Lecture 41 Derived Column Transformation
Lecture 42 Raw to Cleansed Data Flow Requirement
Lecture 43 Raw to Cleansed Data Flow Walkthrough
Lecture 44 Partitioned Datasets
Lecture 45 Filter and Sort Transformations
Lecture 46 Aggregate Transformation
Lecture 47 Join Transformation
Lecture 48 Conditional Split and Union Transformations
Lecture 49 Cleansed to Structured Data Flow Requirement
Lecture 50 Cleansed to Structured Data Flow Walkthrough
Lecture 51 Structured to Analytics Data Flow Requirement
Lecture 52 Structured to Analytics Data Flow Walkthrough
Lecture 53 Links and Resources
Section 6: Scheduling and Chaining Pipelines for Execution
Lecture 54 Execute Pipeline Activity
Lecture 55 Schedule Triggers
Lecture 56 Storage Events Trigger
Lecture 57 Tumbling Window Triggers
Lecture 58 Links and Resources
Section 7: Control Flow, Parameters and Variables
Lecture 59 Data Lake Preparation
Lecture 60 Delete Activity
Lecture 61 Get Metadata Activity
Lecture 62 Lookup Activity
Lecture 63 If Condition and Switch Activities
Lecture 64 Introduction to Parameters in ADF
Lecture 65 Introduction to Variables in ADF
Lecture 66 ForEach Activity
Lecture 67 Metadata Based Approach (Example)
Lecture 68 Course Project Activity: Dynamic Datasets and Folder Structure
Lecture 69 Links and Resources
Section 8: Azure SQL Database for Structured Data
Lecture 70 Section Overview
Lecture 71 Creating our Azure SQL Resource
Lecture 72 Creating our Schema and Tables
Lecture 73 Adding a Unique Identifier Column to Structured Orders Data
Lecture 74 Creating Linked Service and Datasets for SQL
Lecture 75 Important Note Regarding the Dynamic Datasets
Lecture 76 Copy Activity with SQL Sink
Lecture 77 Data Flow with SQL Sink
Lecture 78 Course Project Activity: Storing our Structured Data in the SQL Database
Lecture 79 Links and Resources
Section 9: Tumbling Window Triggers
Lecture 80 Tumbling Window Triggers Overview
Lecture 81 Ingest Data for Specific Time Intervals
Lecture 82 Tumbling Window Trigger Dependencies
Lecture 83 Links and Resources
Section 10: Authentication and Access
Lecture 84 Overview
Lecture 85 Azure Key Vault
Lecture 86 System Assigned Managed Identities
Lecture 87 User Assigned Managed Identities
Lecture 88 Service Principals
Lecture 89 Fine Grained Access Control for Storage Accounts
Lecture 90 Links and Resources
Anyone interested in Azure or Cloud Data Engineering,Anyone interested in working with Big Data,Anyone working with cloud data