What is the “Cloud”?

Why cloud services

Types of cloud models

Deployment Models

• Private Cloud deployment model

• Public Cloud deployment model

• Hybrid cloud deployment model

Characteristics of cloud computing

• On-demand self-service

• Multi-tenancy and resource pooling

• Measured service

Cloud Data Warehouse Architecture

• Shared Memory architecture

• Shared Disk architecture

• Shared Nothing architecture

Core Azure Architectural components

Core Azure Services and Products

Azure solutions

Azure management tools

DWH Life Cycles

• DWH Approaches (INMON and KIMBALL)

• Data Granularity, Data movement stages

Azure subscriptions

Planning and managing costs

Azure support options

Azure Service Level Agreements (SLAs)

Service Lifecycle in Azure

Introduction Azure SQL Database

Comparing Single Database

Managed Instance

Creating and Using SQL Server

Azure SQL Database Tools

Migrating on premise database to SQL Azure

Purchasing Models

• DTU service tiers

• vCore based Model

• Server-less compute tier

Service Tiers

• General purpose / Standard

• Business Critical / Premium

• Hyperscale

Deployment of an Azure SQL Database

What is SQL elastic pools

• Choosing the correct pool size

• Creating a New Pool

• Manage Pools

Monitoring and Tuning Azure SQL Database

Configure SQL Database Auditing

Export and Import of Database

Storage Service and Account

Creating a Storage Account

Standard and Premium Performance

Understanding Replication

Hot, Cold and Archive Access Tiers

Working with Containers and Blobs

Types of Blobs

• Block Blobs

• Append Blobs

• Page Blobs

Blob Metadata

Soft Delete

Azure Storage Explorer

Access blobs securely

• Access Key

• Account Shared Access Token

• Service Shared Access Token

• Shared Access Policy

Storage Service Encryption

Azure Key Vault

What is Data Factory?

Data Factory Key Components

• Pipeline

• Activity

• Linked Service

• Data Set

• Integration Runtime

• Triggers

• Data Flows

Page Blobs

Integration Runtime Provision Required Azure Resources

• Create Resource Group

• Create Storage Account

• Provision SQL Server and Create Database

• Provision Azure Data Factory (ADF) Service

Understanding Data Factory UI

Copy Data from Blob Storage to SQL Database

Copy data from storage account to storage account

Create Linked Service and Create Dataset

Create Pipeline – Integration Service

Copy Data from on-premise SQL Server to Blob Storage

Working with Activities

• Understanding Lookup Activity

• Understanding For Each Activity

• Filter Activity

• Get Metadata Activity

Azure Lift and Shift

Provisioning Azure – SSIS Integration Runtime

Execute SSIS Packages from Azure

Execute SSIS Packages from SSISDB

Triggers

Monitoring Pipeline

• Debug Pipeline

• Trigger pipeline manually

• Monitor pipeline

• Trigger pipeline on schedule

ADF Introduction

Important Concepts in ADF

Create Azure Free Account for ADF

Integration Runtime and Types

• Integration Runtime in ADF – Azure IR

Create Your First ADF

Create Your First Pipeline in ADF

Azure Storage Account Integration with ADF

Copy multiple files from blob to blob

Filter activity – Dynamic Copy Activity

Get File Names from Folder Dynamically

Deep dive into Copy Activity in ADF

• Copy Activity Behavior in ADF

• Copy Activity Performance Tuning in ADF

Validation in ADF

• Get Count of files from folder in ADF

• Validate copied data between source and sink in ADF

Azure SQL Database integration with ADF

• Azure SQL Databases – Introduction to Relational Databases

• Creating Your First Azure SQL Database

• Deployment Models

• Purchasing Modes

Overwrite and Append Modes in Copy Activity

Full Load in ADF

Copy Data from Azure SQL Database to BLOB in ADF

Copy multiple tables in bulk with Lookup & ForEach in Data Factory

Logging and Notification – Azure Logic Apps

• Log Pipeline Executions to SQL Table using ADF

• Custom Email Notifications – Send Error Notification with Logic App

Use Foreach Loop Activity to Copy Multiple Tables – Step-by-Step Explanation

Incremental Load in ADF

• Incremental or Delta Load from SQL to Blob Storage

• Multi-table Incremental Load or Delta Load

• Incrementally Copy New and Changed Files Based on Last Modified Date

Azure Key Vault Integration with ADF

• Secure secrets, keys & certificates in Azure Data

ADF Triggers

• Event-Based Trigger

• Tumbling Window Trigger – Dependency & Parameters

• Schedule Trigger

Self-Hosted Integration Runtime

• Copying On-Premise Data Using Azure Self-Hosted IR

• Data Migration from On-Premise SQL Server to Cloud

• Load Data from On-Premise SQL Server to Azure SQL DB

• Data Migration with PolyBase and Bulk Insert

• Copy Data from SQL Server to Azure SQL DW with PolyBase & Bulk Insert

• Data Migration from On-Premise File System to Cloud

• Copy Data from On-Premise File System to ADLS Gen2

Copying Data from REST API using ADF

• Loop Through REST API – Copy Data to ADLS Gen2

• Linked Service Parameters

AWS S3 Integration with ADF

• Migrate Data from AWS S3 Buckets to ADLS Gen2

Activities in ADF

• Switch Activity – Move and Delete Data

• Until Activity – Parameters & Variables

• Copy Recent Files from Blob Input to Blob Output Folder Without LPV

Snowflake Integration with ADF

• Copy Data from Snowflake to ADLS Gen2

• Copy Data from ADLS Gen2 to Snowflake

Azure CosmosDB Integration with ADF

• Copy Data from Azure SQL DB to CosmosDB

• Copy Data from Blob to CosmosDB

Advanced Concepts in ADF

• Nested ForEach – Pass Parameters from Master to Child Pipeline

• High Availability of Self-Hosted IR & Sharing IR with Other ADFs

Data Flows Introduction

• Azure Data Flows Introduction

• Setup Integration Runtime for Data Flows

• Basics of SQL Joins for Azure Data Flows

• Joins in Data Flows

• Aggregations and Derive Column Transformations

• Advanced Join Transformations with Filter and Conditional Split

• Data Flows – Data Processing Use Case 1

• Restart Data Processing from Failure

• Remove Duplicate Rows & Store Summary Credit Stats

• Difference Between Join vs. Lookup Transformation & Merge Functionality

• Dimensions in Data Flows

• Slowly Changing Dimension Type 1 (SCD1) with HashKey Function

• Flatten Transformation

• Rank, Dense_Rank Transformations

• Data Flows Performance Metrics and Parameters

• How to Use Pivot and Unpivot Transformations

• Data Quality Checks and Logging Using Data Flows

Batch Account Integration with ADF

Custom Activity in ADF

Azure Functions Integration with ADF

Azure Databricks Integration with ADF

ADF Integration with Azure Databricks

Azure Data Lake Analytics Integration with ADF

Introduction to Databricks

Azure Databricks Architecture

Azure Databricks Main Concepts

Databricks Integration with Azure Blob Storage

• Read Data from Blob Storage

• Create Blob Mount Point

Features of Python

Python Virtual Machine (PVM)

Frozen Binaries

Memory Management in Python

Garbage Collection in Python