Azure Data Engineer Course with Real-Time Projects Day2

Azure Data Engineer Course with Real-Time Projects – Live Training

(Industry-oriented training covering Azure SQL, Azure Data Factory, Databricks, Spark, Delta Lake, Kafka, and modern Lakehouse architecture)

The Azure Data Engineer Course with Real-Time Projects is a comprehensive, hands-on live training program designed to help you become an industry-ready Azure Data Engineer. This course takes you from core data engineering fundamentals to building modern, scalable Lakehouse architectures on Azure, using real-world projects and enterprise-grade tools.

You will gain deep expertise in Azure Data Factory, Azure SQL, Azure Storage, Databricks, Apache Spark, Delta Lake, and real-time data streaming with Kafka and Azure Event Hubs. The program emphasizes practical implementation, covering end-to-end data pipelines, batch and streaming ingestion, transformations, orchestration, governance, and performance optimization.

Throughout the course, you will work on multiple real-world case studies and projects, including data modeling, ETL/ELT pipelines, incremental data processing, Slowly Changing Dimensions (SCD), and Lakehouse implementation using Databricks and Unity Catalog. You will also learn DevOps integration, monitoring, alerting, and automation, ensuring your solutions meet enterprise standards.

By the end of this program, you will be confident in designing, building, and managing secure, scalable, and high-performance data platforms on Azure, making you job-ready for Azure Data Engineer roles in today’s data-driven organizations.

About the Instructor:

Ashok is a seasoned Data Engineer with deep expertise in building scalable, reliable, and production-grade data pipelines using Azure and big data technologies. With multiple Azure certifications and real-world experience, he has worked across on-premises and cloud environments to migrate, optimize, and maintain robust data infrastructures.

He is passionate about simplifying complex systems and teaching others how to turn data into actionable insight. Through this bootcamp, Ashok brings his industry learnings, hands-on practices, and troubleshooting techniques to guide learners from foundational understanding to real-world project implementation.

Over the years, he has successfully completed 20+ batches and trained over 400+ students across different domains, helping many of them transition into data engineering roles and advance their careers.

Sample videos:

Azure Data Engineer Course with Real-Time Projects Live Training – Demo

Azure Data Engineer Course with Real-Time Projects Live Training – Day1

Live Sessions Price:

For LIVE sessions – Offer price after discount is 3~~00 USD 259 ~~USD~~~~ 109 USD Or ~~13000 INR~~ ~~12900 INR~~ 8900 Rupees

Enroll For Free Demo

Weekend Batch

Free Day2 Session:

14th February @ 10:00 AM – 11:00 AM (IST) (Indian Timings)

13th February @ 11:30 PM – 12:30 AM (EST) (U.S Timings)

14th February @ 4:30 AM – 5:30 AM (BST) (UK Timings)

Class Schedule:

For Participants in India: Saturday & Sunday @ 10:00 AM – 12:00 PM (IST)

For Participants in the US: Friday & Saturday @ 11:30 PM – 1:30 AM (EST)

For Participants in the UK: Saturday & Sunday @ 4:30 AM – 6:30 AM (BST)

Trainer Leave Notice

Please note that the trainer has pre-planned leave on the following dates:

21st February
22nd February
7th March
8th March

What student’s have to say about Trainer :

👩 Fatima Begum:
Trainer Ashok made Azure concepts crystal clear. His step-by-step guidance in ADF really helped me. The capstone project gave me real job-level experience.

👨 Karthik Reddy:
Excellent sessions! The real-world projects gave me confidence to face interviews. Ashok is very patient and supportive. His way of teaching Databricks is amazing. Even tough topics felt simple with his examples.

👨 Daniel George:
Ashok guided us through Azure, SQL, Databricks, and ADF step by step. Even tough concepts became easy. Truly an amazing trainer!

👩 Neha Kulkarni:
Loved the Azure sessions. Dashboards were very practical to business use cases. Ashok explains concepts with real projects.

👩 Areeba:
Ashok’s way of teaching is very clear and simple. Azure Data Factory labs boosted my confidence. The mini project was so useful—it felt like real industry experience.

What will I learn by the end of this course?

Understand core data engineering fundamentals, data types, processing patterns, and modern data platform concepts
Learn the evolution from Data Warehouses and Data Lakes to Lakehouse architecture and when to use each
Design and implement relational and dimensional data models (OLTP, OLAP, Star & Snowflake schemas)
Build, schedule, and monitor end-to-end ETL/ELT pipelines using Azure Data Factory (ADF)
Work confidently with Azure Storage (ADLS, Blob Storage), access tiers, lifecycle management, and optimization
Develop strong skills in Azure SQL, including CTEs, indexes, views, and stored procedures
Implement data transformations using ADF Data Flows and PySpark
Process real-time streaming data using Kafka, Azure Event Hubs, and Stream Analytics
Master Apache Spark and PySpark, including transformations, actions, jobs, stages, and performance concepts
Build and manage Delta Lake tables with ACID transactions, schema evolution, time travel, and MERGE operations
Implement Lakehouse architecture using Azure Databricks, notebooks, workflows, and job automation
Apply data governance and security using Unity Catalog, access controls, lineage, and data masking
Design incremental data pipelines and SCD implementations using Databricks and Lakeflow pipelines
Create data warehouses using Databricks SQL, views, dashboards, and SQL alerts
Integrate Azure DevOps, Logic Apps, and monitoring alerts for pipeline automation and reliability
Deliver real-world, end-to-end projects that make you job-ready as an Azure Data Engineer

Who can enroll for this course?

Fresh graduates looking to start a career in Data Engineering or Cloud Technologies
Software engineers and developers who want to transition into Azure Data Engineering roles
ETL / Data Warehouse professionals aiming to modernize their skills with Azure, Databricks, and Lakehouse architecture
Data analysts and BI professionals who want to move into end-to-end data engineering
Cloud engineers seeking hands-on experience with Azure data services and pipeline development
Database professionals (SQL Developers / DBAs) looking to expand into cloud-based data platforms
Big data professionals wanting to strengthen skills in Spark, PySpark, Delta Lake, and Databricks
IT professionals planning a career shift into high-demand Azure Data Engineer roles
Students and working professionals preparing for real-world projects and industry use cases
Anyone with a basic understanding of SQL, programming concepts, or cloud fundamentals who wants to become job-ready in Azure Data Engineering

Salient Features:

48+ Hours of Live Training along with recorded videos
Lifetime access to the recorded videos
Course Completion Certificate

Course syllabus:

Day 0: Data fundamentals

Discuss data terminologies and how they impact data storage
- Compute (CPU)
- Memory (RAM, Disk)
- Storage (SSD, HDD)
- Data Caching
- Throughput
- Latency
- Network Bandwidth
- Batch processing
- Real time processing
- Pipelines
- Orchestration
Types of Data:
- Unstructured
- Structured
- Semi – structured

Week 1: Building Blocks: From DW to Lakehouse

Day 1: From Warehouse to Lakehouse: A Guide to Modern Data Platforms
- Characteristics of database, data warehouse, data lake, data Lakehouse
- Differences between data warehouse and data lake
- Limitations of data warehouse and data lake
- Why data Lakehouse is preferred over data warehouse / data lake
- Demo: Architecture diagrams of DWH, DL, DLH
- ETL vs ELT which is preferred?
- Demo: ELT architecture diagram walkthrough and ETL architecture diagram walkthroug
- Conceptual, Logical and Physical data modelling
- Relational data modelling and its advantages
- Project 1: Design a relational data model for retail

Day 2: Data Modelling – OLAP
- Dimensional data modelling advantages
- Introduction to fact and dimension tables
- Star and snowflake schema design
- Project 2: Design a dimensional data model
- Introduction to Slowly changing dimensions
- Types of SCD
- Implementing SCD types in Dimensional data model

Week 2: Cloud Mastery: Storage C Access

Day 3: Fundamentals of cloud and azure
- CapEx vs OpEx
- Advantages of cloud over on premises
- Disadvantages of data centers
- Types of cloud:
  - Public
  - Private
  - Hybrid
- Azure Portal Walkthrough
- Azure Terminologies:
  - Resource groups
  - Subscriptions
  - Entra ID
  - IAM roles
  - Cost Management
- Data redundancy options:
  - Data centers
  - Availability zones
  - Regions
- How Azure ensures data availability using regional architecture
- Introduction to Azure Storage:
  - Distributed storage and its advantages
  - Azure blob storage vs Azure data lake storage
  - Access types in Azure storage account
  - Types of storage services provided in azure storage account:
    - Containers
    - File Shares
    - Tables
    - Ǫueues
- Demo: Folder creation, file uploads, file deletion and storage account options

Day 4: Advance concepts in Azure Storage:
- Types of blobs:
  - Block blob
  - Page blob
  - Append blob
- Types of data access tiers:
  - Hot
  - Cool
  - Archive
- Versioning
- Soft delete
- Lifecycle management rules
- Project 3: Azure Storage Optimization project

Week 3: SǪL C Pipelines: Your First ETL

Day 5: A Guide to Azure SǪL: Databases, Servers, and Rule Management
- Azure SǪL family introduction
- Services offered in Azure SǪL
- Advantages of using Azure SǪL database
- Create and configure SǪL server and SǪL database in Azure
- Setup rules to allow access (firewall management)
- Project 4: Solve a Case study using Azure SǪL
- Assignment 1: Pizza Runner Case Study

Day 6: Azure Data Factory Basics: What It Is and Why It Matters
- What is Azure Data Factory and advantages
- How Azure Data factory solves some of the data movement problems in the industry
- Connecting azure storage account with ADF
- Demo: Creating Azure data factory data studio
- Azure Data Factory Terminologies
  - Integration Runtime (SHIR, Azure IR, Linked SHIR)
  - Demo:
    - Create SHIR (on premises)
    - Create Linked SHIR
    - Create Azure IR
  - Linked services
  - Demo: create Linked service for:
    - Azure Storage account
    - Git Hub Repo
    - Azure Databricks
    - Azure SǪL
    - On premises SǪL Server
    - On premises file system
  - Datasets
    - Data Stores
    - Compute Stores
  - Pipelines
  - Triggers
    - Scheduled
    - Event based
    - Tumbling window

Week 4: Pipeline Development: Azure Data Factory

Day 7: Understanding Azure Data Factory: Activities, Parameters, Variables, and Authentication
- Activities:
  - Copy data
  - Data flow
  - Get Metadata
  - Lookup
  - Execute pipeline
  - For Each
- Activity dependencies:
  - On success
  - On skip
  - On Failure
  - On Complete
- Parameters and how to dynamically parameterize ADF pipelines
- Variables and how to use them in a pipeline
- Differences between variables and pipelines
- Demo: Prepare a pipeline with the Get Metadata activity and explain the variables and pipelines
- Azure Key Vault:
  - Features of Key Vault
  - Keys, Secrets, Certificates in key vault
  - How to authorize a user in Key vault using IAM roles
  - Demo: Connect to Azure storage account using key vault
- Project 5: Harmonizing Clinical C Real-World Data at Fusion Pharma Analytics
Day 8: Azure Data Factory: Transformations using Data Flows
- When should we use Data Flows and why
- Demo: Creating data flows in pipelines and connecting data
- Data Flow activities:
  - Conditional split
  - Source Stream
  - Sink stream
  - Assert
  - Derived column
  - Select
  - Alter row
- Project 6: Implementing SCD types using ADF Data flows

Week 5: Designing and Automating Data Pipelines with Azure Data Factory

Day 9: How Azure DevOps Enhances Azure Data Factory Workflows
- Git Integration in ADF
- Rules to consider while triggering pipelines with Git integration
- Live mode vs Git mode in ADF pipelines
- Collaboration and main branches in ADF
- How DevOps enhances ADF pipeline versioning
- Demo: Connecting ADF with Azure DevOps and creating git branches in ADF for collaboration
- Project 7: Sea Freight Logistics Data Modernization
Day 10:
- Project 8: Sea Freight Logistics Data Modernization
- Introduction to Azure Logic Apps
  - Overview
  - How logic apps can be connected to ADF
  - Actions C Triggers
  - Demo: Connecting Azure Logic Apps with ADF workspace
- Pipeline Monitoring and Alerts:
  - Add Alerts in the pipeline
  - Send notification emails to stake holders through Logic Apps
  - Re-run from the last processed activity
  - Review logs in copy activity
  - Update pipeline status in SǪL table

Week 6: SǪL Deep Dive and Introduction to Apache Kafka

Day 11: SǪL Building Blocks: CTEs, Views, Stored Procedures, and Indexes
- How to write CTEs in SǪL server
- Difference between CTE and Subquery
- What are Indexes
- Advantages and disadvantages of using Indexes
- Types of Indexes:
  - Clustered
  - Non-clustered
- What are stored procedures
- When should we use stored procedures
- Demo: Creating a stored procedure in SSMS
- Project 9: Architecting the Operational Database for Aura Music Festival
Day 12: Streaming Data using Azure Event Hubs, Kafka
- Introduction to Kafka
- Terminologies:
  - Topics
  - Partitions
  - Producers
  - Consumers
  - Brokers
- Serialization and de-serialization in Kafka
- Which file format is best suited for Kafka messaging
- Azure Event Hubs C Stream Analytics:
  - Introduction to Azure event hubs and advantages
  - Integrate Kafka with event hubs for real time streaming of data
  - Use stream analytics jobs in event hubs
- Project 10: Ingest real -time streaming data to ADLS using Event Hubs

Week 7: Getting Started with Apache Spark

Day 13: Mastering Pandas Data Frame Operations
- Pandas Data frame and Series
- Project 11: Comparing Online In-Store Product Performance
- Getting Started with Apache Spark: Distributed Computing and Architecture
  - What is distributed computing
  - Explain Apache spark and how it is faster than Map Reduce in Hadoop
  - Advantages of spark:
    - DAG scheduler
    - Lazy evaluation
    - Parallel processing
    - In-memory computation
  - Architecture of spark and it’s internals
  - Demo: Connect spark in Jupyter notebook
Day 14: Components of Apache spark:
- Driver
- Executor
- Resource manager
- Spark session
- Data Frame
- Dataset
- RDD
- Demo: Create RDD and Data frame in notebook
- Reading and Writing data using Pyspark:
  - DataFrame Reader API
  - DataFrame Writer API
  - Read options for each file format
  - Read modes (Permissive, Fail Fast, Malformed)
  - How to process corrupt / bad data
  - Demo: Read and write CSV data
  - Demo: Process corrupt records
  - Spark SǪL, Temporary views
  - Write SǪL queries on data frame
  - Temporary Views
  - Global temporary views
  - Difference between temporary views and global temp view

Week 8: Pyspark continued, Intro to Azure Databricks

Day 15: Transformations, Actions in PySpark
- Narrow transformations
- Wide transformations
- Actions and dependencies with transformations
- Project 12: Transforming big mart sales
- Assignment 2: Case study on Diner’s pizza using pyspark
- Jobs, Stages and tasks
- Data shuffle (shuffle partitions, shuffle read, shuffle write)
- Demo: Analyze jobs, stages and tasks required based on a scenario

Day 16: Introduction to Azure Databricks
- Databricks Architecture, advantages
- Terminologies:
  - Compute
    - All purpose
    - Job
    - Serverless
  - Workspaces
    - Repos
    - Shared
    - Users
  - DBSǪL Warehouse
    - Serverless
    - General
  - Data Ingestion feature
    - Lake flow Connect
    - External connectors (FiveTran)
    - Upload data to delta table
    - Upload to a volume
  - Demo: Portal Walkthrough, Create resources
  - Connect Databricks to Visual Studio Code
  - Explore available options in VS Code
  - How Databricks works with Azure:
    - ARM Templates
    - Managed resource group walk through
  - Demo: Architecture diagram of how Databricks works with Azure

Week 9: Lakehouse Implementation: Notebooks to Delta Lake

Day 17: Meta store and Unity catalog
- Introduction to Unity Catalog
- Data governance using UC
- Three level namespace model
- Permission model in UC
- Features of UC:
  - Storage credentials
  - External locations
  - Data Lineage
  - History
  - Data Insights
- Demo: Create Meta store with external location
- Types of Catalogs:
  - Standard
  - Foreign (Lake house federation)
- Schemas (External, managed)
- Tables (External, managed)
- Views (Temporary, materialized)
- Volumes (external, managed)
- Demo:
  - Create storage credential for azure storage
  - Access files in ADLS directly in
  - Create catalog, schema and delta tables on top of ADLS
  - Upload csv data to volume and access it in
  - Create Views that can be used in Dashboards
Day 18: Lakehouse Design and Workflow Automation
- Discuss Lakehouse architecture and how it is implemented in industry
  - Three-layer architecture
- Introduction to notebooks
- How notebook automation works in Azure Databricks
- Demo: Create workflow along with job cluster and schedule notebook execution
- Project 13 – Implement Lakehouse architecture using ADF and Databricks

Week 10: Azure Databricks Continued

Day 19: Process Data in Databricks using PySpark
- Access ADLS within Databricks using service principals
- Secret scopes, Databricks utilities walkthrough
- Transform dataset using both SǪL and Pyspark in Databricks
- Cluster Pools
- Serverless clusters
- Cluster policies
- Demo: create your own cluster policy
- Limitations of parquet format
- Introduction to delta lake tables
- Parquet vs Delta Lake tables
Day 20: Delta Lake, Delta tables
- Unique features of Delta Lake Tables using sample data:
  - Schema evolution
  - ACID compliance
  - Time travel
  - Metadata log
- Deep vs shallow clones
- MERGE and UPSERTS
- Deletion Vectors
- VACCUUM
- Delta table properties
- copy into, Autoloader
- Demo: Process data using MERGE, Copy INTO, Autoloader and understand the differences

Week 11: Advanced Lakehouse: Incremental Patterns and DLT

Day 21: Lake flow declarative pipelines
- Project 14: Incremental data ingestion using Databricks
- Project 15: Implementing SCD types in Databricks

Day 22: Declarative Pipelines
- Imperative vs Declarative pipelines
- How they differ from classic notebooks-based Spark
- Lake flow declarative pipelines overview
- Demo: Lake flow declarative pipeline for ETL
- Joins, aggregations, window functions in Python and SǪL within declarative
- Data quality with expectations for incremental loads: defining constraints, severity (warn/fail), and handling bad
- Running and scheduling pipelines:
  - Triggers
  - Full vs incremental updates
  - Integrating with jobs and

Week 12: Data Warehousing using Databricks

Day 23: Data warehousing using DB SǪL
- SǪL warehouses and types
- Portal walkthrough
- Demo: Creating a Warehouse
- Setting up SǪL alerts
- DB SǪL commands
- Functions in Unity Catalog using DB SǪL
  - Row filter
  - Column Mask
  - Applying tags to the data for data classification
  - Demo: Apply row filters, column masks on sample data
- Users and Groups
  - Create users and groups in Databricks
  - Assign roles based on user or group
  - Allow access to data based on user or group

Day 24: Views in DB Warehouses
- Types of views
- Advantages, differences between types of views
- Demo: create all types of views
- Project 16: Build a sales dashboard (e.g., by region/product) using existing Lakehouse tables

Bonus:

Kahoot sessions for important topics to challenge
Assignments as a homework for important topics
eBooks for free
Doubt-clearing sessions on

Events

Azure Data Engineer Course with Real-Time Projects Day2

Event Information

Azure Data Engineer Course with Real-Time Projects – Live Training

Tricentis Tosca – Automation Training Day 2

Performance Testing using NeoLoad Live Training Day 2

Buy Ticket