Azure Cloud Data Engineering & Migration: Master Data Platforms, Pipelines & Transformation – Live Training
(Learn to design and build scalable data pipelines on Azure. Master Data Factory, Databricks, and Lakehouse architecture with real-world projects. A complete 30-day bootcamp to become industry-ready in Azure Data Engineering)
The Azure Data Engineering & Migration: Data Factory, Databricks & Lakehouse is a structured 30-day hands-on program designed to take you from the foundations of modern data architecture to building real-world, end-to-end data solutions on Azure.
You will learn how to design scalable data pipelines, migrate on-premise data to the cloud, and work with core Azure services including Data Factory, Databricks, Azure SQL, and Storage Accounts. Along the way, you’ll master SQL fundamentals, Spark-based computing, and Lakehouse architecture with Delta Tables and Unity Catalog.
By the end of this course, you’ll be equipped to build and automate enterprise-grade data pipelines, transform data at scale, and implement modern analytics solutions—making you industry-ready as an Azure Data Engineer.
About the Instructor:
|
Ashok is a seasoned Data Engineer with deep expertise in building scalable, reliable, and production-grade data pipelines using Azure and big data technologies. With multiple Azure certifications and real-world experience, he has worked across on-premises and cloud environments to migrate, optimize, and maintain robust data infrastructures. He is passionate about simplifying complex systems and teaching others how to turn data into actionable insight. Through this bootcamp, Ashok brings his industry learnings, hands-on practices, and troubleshooting techniques to guide learners from foundational understanding to real-world project implementation. Over the years, he has successfully completed 20+ batches and trained over 400+ students across different domains, helping many of them transition into data engineering roles and advance their careers. |
Live Sessions Price:
For LIVE sessions – Offer price after discount is 300 USD 259 99 USD Or USD13000 INR 12900 INR 7900 Rupees
OR
Free Demo Session:
8th October @ 7:30 AM – 8:30 AM (IST) (Indian Timings)
7th October @ 10:00 PM – 11:00 PM (EST) (U.S Timings)
8th October @ 3:00 AM – 4:00 AM (BST) (UK Timings)
Class Schedule:
For Participants in India: Monday to Friday @ 7:30 AM – 8:30 AM (IST)
For Participants in the US: Sunday to Thursday @ 10:00 PM – 11:00 PM (EST)
For Participants in the UK: Monday to Friday @ 3:00 AM – 4:00 AM (BST)
What student’s have to say about Trainer :
|
👩 Fatima Begum: 👨 Karthik Reddy: 👨 Daniel George: 👩 Neha Kulkarni: 👩 Areeba: |
What will I learn by the end of this course?
- Understand the foundations of modern data architecture (warehouses, lakes, and Lakehouse).
- Design and build scalable data pipelines using Azure Data Factory (ADF).
- Work with Azure SQL, Storage Accounts, and Access Management effectively.
- Master ETL processes, orchestration, and automation in real-world scenarios.
- Gain hands-on experience with Databricks, PySpark, and Delta Lake for big data transformation.
- Implement Lakehouse architecture with Unity Catalog and Delta Tables.
- Learn SQL fundamentals to advanced queries, along with Python Pandas for data manipulation.
- Build and deploy end-to-end data migration projects from on-premises to Azure Cloud.
- Integrate Azure DevOps, Logic Apps, and Event Hubs for automation and real-time streaming.
- Become job-ready as an Azure Data Engineer with project-based, industry-aligned skills.
Who can enroll for this course?
- Aspiring Data Engineers looking to break into the field with hands-on Azure and Databricks skills.
- Software Developers or Testers aiming to transition into data engineering roles.
- Data Analysts / BI Professionals who want to move from reporting to end-to-end data pipeline development.
- Fresh Graduates or Students eager to learn cloud-based data engineering with real-world projects.
- IT Professionals seeking to upskill in Azure, Spark, and Lakehouse architecture for career growth.
Salient Features:
- 30+ Hours of Live Training along with recorded videos
- Lifetime access to the recorded videos
- Course Completion Certificate
Course syllabus:
Week 1: Foundations of Modern Data Architecture
- Day 1: Demystifying System Design: Key Concepts of Client-Server Architecture
- What is client-server architecture
- How is data maintained in an organization (MNCs)
- Discuss about Data centers (on-prem), databases.
- What happens when we open a file in our computer (RAM vs Disk)
- Day 2: Why On-Prem Data Storage Gets Complicated
- Challenges of storing data in a single datacenter
- Single point of failure
- Scaling
- Data Availability
- Security
- Replication
- Cost
- Performance
- Discuss data terminologies and how they impact data storage
- Compute (CPU)
- Memory (RAM, Disk)
- Storage (SSD, HDD)
- Data Caching
- Throughput
- Latency
- Network Bandwidth
- Batch processing
- Real time processing
- Pipelines
- Orchestration
- Challenges of storing data in a single datacenter
- Day 3: Understanding Data: Types, Sources, and Formats
- Types:
- Unstructured Data
- Structured Data
- Semi – structured data
- Sources:
- APIs
- Flat files
- Tables (JDBC)
- Streams
- Formats:
- CSV
- JSON
- Avro
- Parquet
- Types:
- Day 4: From Warehouse to Lakehouse: A Guide to Modern Data Platforms
- Characteristics of database, data warehouse, data lake, data Lakehouse
- Differences between data warehouse and data lake
- Limitations of data warehouse and data lake
- Why data Lakehouse is preferred over data warehouse / data lake
- Day 5: Why Cloud Migration Matters: Balancing CapEx and OpEx
- What is capital and operational expenditure
- Analogy: buy vs rent for capital and operational expenditures
- What is the need of cloud data migration
- How cloud data migration saves cost and improves performance
- Types of cloud:
- Public
- Private
- Hybrid
- How modern data systems are built on cloud
Week 2: Azure Services Deep Dive—Access, Storage, and Data Factory
- Day 6: Azure Fundamentals: Terminologies, Data Centers, and Regional Architecture
- Why Azure
- Azure Terminologies:
- Resource groups
- Subscriptions
- Entra ID
- IAM roles
- Cost Management
- Azure portal navigation – Demo
- How Azure ensures data availability using regional architecture (data centers, availability zones, regions)
- Day 7: A Guide to Azure Access Types: RBAC, Managed Identities, and SAS
- Different access types provided by Azure
- Role based access control
- Access key and its limitations
- SAS token and its limitations
- Managed identity (System assigned, User assigned)
- Service principal and its limitations
- Which access type is preferred in the industry and why
- Day 8: Azure Storage Explained: Why Distributed Systems Matter
- Distributed storage and its advantages
- Azure storage account walkthrough
- Azure blob storage vs Azure data lake storage
- Types of storage services provided in azure storage account:
- Containers
- File Shares
- Tables
- Queues
- Types of blobs:
- Block blob
- Page blob
- Append blob
- Types of data access tiers:
- Hot
- Cool
- Archive
- Versioning
- Soft delete
- IAM roles to access azure storage in any other service
- Use case of Blob Storage and Data Lake storage
- Day 9: A Guide to Azure SQL: Databases, Servers, and Rule Management
- Azure SQL family introduction
- Services offered in Azure SQL:
- Advantages of using Azure SQL database
- Create and configure SQL server and SQL database in Azure
- Setup database and server level rules to allow access
- Hands on: Solve 5 SQL queries with explanation in Azure SQL
- Connect Azure SQL to Azure Data Studio and create SQL notebooks
- Day 10: Azure Data Factory Basics: What It Is and Why It Matters
- What is Azure Data Factory
- Advantages of Azure Data Factory
- How Azure Data factory solves some of the data movement problems in the industry
- Definition of ETL and how Azure data factory achieves ETL with high scalability
- Azure data factory portal navigation
- Azure Data Factory terminologies:
- Integration Runtime (SHIR, Azure IR, Linked SHIR)
- Linked services
- Datasets
- Pipelines
Week 3: Designing and Automating Data Pipelines with Azure Data Factory
- Day 11: Understanding Azure Data Factory: Activities, Parameters, Variables, and Authentication
- Activities:
- Copy data
- Data flow
- Get Metadata
- Lookup
- Execute pipeline
- For Each
- Parameters and how to dynamically parameterize ADF pipelines
- Variables and how to use them in a pipeline
- Differences between variables and pipelines
- How to authenticate other services within ADF using Key vault
- Activities:
- Day 12: Data Migration with Azure Data Factory: Building a Sample Pipeline
- Creating Azure SQL Database and sample data
- Creating Azure storage account and containers with sample data
- Create a sample pipeline to copy data from ADLS to Azure SQL
- Day 13: Azure Data Factory: Automating Pipelines with Triggers and Data Flows
- Difference between Debug and a Trigger
- Rules to trigger an ADF pipeline
- Types of Triggers:
- Event based
- Scheduled Trigger
- Tumbling Window
- Custom trigger
- Demo: Creating a trigger and scheduling the pipeline
- When should we use Data Flows and why
- Data Flow activities:
- Conditional split
- Source Stream
- Sink stream
- Assert
- Derived column
- Select
- Alter row
- Demo: Performing data flow operations in a sample pipeline before moving to Azure SQL
- Day 14: How Azure DevOps Enhances Azure Data Factory Workflows
- Git Integration in ADF
- Rules to consider while triggering pipelines with Git integration
- Live mode vs Git mode in ADF pipelines
- Collaboration and main branches in ADF
- How DevOps enhances ADF pipeline versioning
- Day 15: End-to-End Data Migration: On-Prem to Azure Cloud (Project)
- Create on-prem SSMS resource with sample data
- Install SHIR and connect to SSMS
- Setup Azure storage account (ADLS)
- Create Linked services for ADLS and on-prem SSMS
- Build a pipeline to copy data from SSMS in the on-prem to ADLS in Azure
- Discuss the common errors, debugging strategies
Week 4: SQL Deep Dive and Introduction to Spark-Based Computing
- Day 16: Getting Started with SQL Server: Fundamentals and Simple Queries
- SQL Server fundamentals and connect to database
- Discuss databases, schema, tables
- How to execute queries in SSMS.
- Discuss about SQL functions available only in SQL server
- DDL, DML operations on tables
- Day 17: Mastering SQL Logic: Joins, Aggregations, and Window Functions
- Types of Joins:
- Left
- Right
- Inner
- Cross
- Self
- Anti
- Semi
- Aggregation functions and when to use them
- Window functions and when to use them
- Analytical functions and when to use them
- Types of Joins:
- Day 18: SQL Building Blocks: CTEs, Views, Stored Procedures, and Indexes
- How to write CTEs in SQL server
- Difference between CTE and Subquery
- What are Indexes
- Advantages and disadvantages of using Indexes
- Types of Indexes:
- Clustered
- Non-clustered
- What are stored procedures
- When should we use stored procedures
- Creating a stored procedure in SSMS
- Day 19: Optimizing SQL Queries and Mastering Pandas Data Frame Operations
- Performance Tuning strategies in SQL
- Introduction to python programming and pandas
- Syntax differences between SQL and pandas
- Demo: Perform data frame operations using pandas
- Day 20: Getting Started with Apache Spark: Distributed Computing and Architecture
- What is distributed computing
- Explain Apache spark and how it is faster than Map Reduce in Hadoop
- Difference between Pandas and Spark in Python
- Architecture of spark and it’s internals
- Components of spark:
- Driver
- Executor
- Resource manager
Week 5: Hands-On with Databricks—From Notebooks to Delta Lake
- Day 21: Understanding PySpark: Memory Handling, Transformations, and Actions
- Deep dive into pyspark memory management
- Performing data transformations using pyspark transformations
- Difference between narrow and wide transformations with examples
- Performing join operations in pyspark
- Difference between transformations and actions
- Demo: Transforming a sample dataset using Pyspark
- Day 22: Getting Started with Databricks: Notebooks and Core Concepts Explained
- What is Databricks
- Architecture of Databricks
- Demo: create azure Databricks service
- Walk through of Databricks portal
- Discuss some of the core concepts:
- Compute
- Catalog
- Warehouses
- Workspaces
- Data Ingestion
- DBFS
- Workflows
- Day 23: Databricks in Action: Clusters, Lakehouse Design, and Workflow Automation
- Discuss Lakehouse architecture and how it is implemented in corporate
- Cluster types:
- All purpose
- Job
- Serverless
- Demo: Create all-purpose cluster in Databricks
- Define Workflows and when should we use them
- How notebook automation works in Azure Databricks
- Demo: Create workflow along with job cluster and schedule notebook execution
- Day 24: Processing Migrated Data in Databricks: PySpark with CSV and Parquet Files
- Access ADLS within Databricks using service principals
- Transform sample dataset using both SQL and Pyspark in Databricks
- Discuss types of views and when should we use them
- Demo: Measuring processing time for CSV and parquet formats
- Why Parquet is best format for big data
- Day 25: Databricks Essentials: Unity Catalog, Delta Tables, and Parquet Pitfalls
- Limitations of parquet format
- Introduction to delta lake tables
- Parquet vs Delta Lake tables
- Unique features of Delta Lake Tables:
- Schema evolution
- ACID compliance
- Time travel
- Metadata log
- Unity Catalog introduction
- Difference between traditional approach vs Unity Catalog
- Demo: Create unity catalog and perform sample transformations in delta lake tables
Week 6: Foundations of Modern Data Architecture
- Day 26: Project Kick-Off: Building a Real-World Data Pipeline from Scratch
- Discuss Case Study
- Build a solution and identify the azure services required
- Prepare a strategy to connect and authenticate these services
- Create the services and sample data in Azure
- Day 27: Project Setup: Deploying Azure Data Factory for Pipeline Development
- Create and configure the ADF pipeline.
- Setup ADLS storage service for source and sink
- Build the pipeline for data movement
- Create Error handling mechanism in the pipeline
- Schedule the pipeline to automatically trigger when a file is uploaded to ADLS
- Day 28: Transforming Data with Spark on Databricks
- Connect ADLS to Azure Databricks
- Read data from ADLS to pyspark dataframes
- Implement Lakehouse architecture
- Create Delta tables and transform data using SQL
- Write the transformed data back in ADLS
- Day 29: Automating Alerts with Azure Logic Apps and Data Factory
- Introduction to Logic Apps
- Setup Logic App alerts to trigger if ADF pipeline fails
- Demo: Walk through of sending emails to users if a pipeline fails
- Day 30: Bonus Session: Ingesting Real-Time Data into ADLS Using Azure Event Hubs
- Intro to Event hubs and Kafka
- Prepare sample data using python for real-time data ingestion
- Configure event hubs and provide sample data
- Create Streaming jobs in event hubs to ingest the data to ADLS
- Read event hub data into Databricks using spark streaming
- Write the streaming data to ADLS from Databricks
Bonus:
- Kahoot sessions for important topics to challenge yourself.
- Assignments as a homework for important topics
- eBooks for free
- Doubt-clearing sessions on demand.
Live Sessions Price:
For LIVE sessions – Offer price after discount is 300 USD 259 99 USD Or USD13000 INR 12900 INR 7900 Rupees
Sample Course Completion Certificate:
Your course completion certificate looks like this….
Important Note:
To maintain the quality of our training and ensure smooth progress for all learners, we do not allow batch repetition or switching between courses. Once you enroll in a batch, please make sure to attend the classes regularly as per the schedule. We kindly request you to plan your learning accordingly. Thank you for your support and understanding.
Typically, there is a one-day break following public sessions.