Azure Data Engineer

Follow us on

Microsoft Certification: Azure Data Engineer Associate (DP-203)

Skills measured in DP-203:

  • Design and implement data storage solutions
  • Design and develop data processing (building and deploying data pipelines)
  • Design and implement data security
  • Monitor and optimize data storage and data processing

Key Skills covered:

  Azure Blob Storage  Azure Cosmos DBAzure SQLAzure Data Lake Storage
  Azure Data Factory  Azure Synapse Analytics  Azure DatabricksAzure Stream Analytics
  Azure Batch  Azure PipelinesAzure MonitorAzure HDInsight

Azure for the Data Engineer

  • Cloud Computing basics
  • Introduction to Microsoft Azure
  • Applications of Azure
  • Azure Services
  • Understanding Data
  • On-premises vs Cloud-based servers
  • Azure Data Engineer roles & tasks
  • Data Engineering processes
  • Use cases for the Cloud

Data Services on Azure Data Platform

  • Data types & categories
  • Understanding Azure Data storage
  • Data storage in Azure Data Lake Storage
  • Azure Cosmos DB
  • Azure SQL database
  • Azure Synapse Analytics
  • Azure Stream Analytics
  • Azure HDInsight
  • Azure Databricks
  • Azure Data Factory

Data Storage in Azure

Non-Relational data stores

  • Document data stores
  • Columnar data stores
  • Key/value data stores
  • Graph data stores
  • Object data stores

Azure Blob storage

Azure Cosmos DB

  • Data Partitioning: Horizontal/Vertical/Functional
  • Consistency levels in Cosmos DB

Non-Relational Database vs NoSQL

Azure Data Lake Storage

  • Why Data Lake?
  • Data Lake architecture
  • Data Lake key components
  • How data lake stores data
  • Azure Data Lake Storage Gen2

Azure SQL Database

  • Deployment models
  • High-Availability models

Azure Database for MySQL

PolyBase

Azure Synapse Analytics (Azure SQL Datawarehouse)

  • SQL Analytics and SQL Pool in Azure Synapse
  • Key components of big data solution
  • MPP architecture

Data Integration with Azure Data Factory (ADF)

  • Data integration patterns
  • ADF components
  • Linked services
  • Datasets
  • Activities and Pipelines
  • Integration runtimes
  • ADF ingestion methods
  • ADF connectors
  • Copy activity
  • ADF Transformation methods
  • ADF Mapping data flow
  • ADF Control flow
  • Working with ADF pipelines
  • Executing data factory packages
  • Monitor ADF pipelines

Data Warehousing Solutions with Azure Synapse Analytics

  • Intro to Azure Synapse Analytics (Azure SQL Datawarehouse)
  • How Azure Synapse Analytics works
  • When to use Azure Synapse Analytics
  • Create Azure Synapse Analytics workspace
  • Synapse Analytics SQL
  • Creating Apache Spark Pools
  • Apache Spark Notebooks
  • Load data in Spark notebooks
  • Load data in Spark DataFrames
  • Integrating SQL and Spark Pools
  • Orchestrating data integrating with Synapse Analytics Pipelines
  • Understanding Synapse Analytics processes
  • Working with Azure Synapse Studio
  • Data hub
  • Develop hub
  • Integrate hub
  • Monitor & Manage hub
  • Designing modern data warehouse architecture components
  • Designing Ingestion patterns
  • Data storage
  • Transforming data with Synapse Analytics
  • Monitoring data warehouse activities in Synapse Analytics
  • Column and row-level security

Data Engineering with Azure Databricks

  • Understanding Azure Databricks
  • Databricks Notebooks
  • Working with Databricks Notebooks
  • Databricks Spark Cluster
  • Architecture of Spark job
  • Read & Write data in Azure Databricks
  • Working with DataFrames in Azure Databricks
  • DataFrames Transformations and Functions
  • Eager and Lazy execution
  • Security & Data protection in Azure Databricks
  • Delta Lakes to create, append data to Apache Spark tables
  • Structured streaming
  • Scheduling Databricks jobs in a data factory pipeline

Large-Scale Data Processing with Azure Data Lake Storage

  • Intro to Data Lake Storage
  • Understanding Data Lake Storage Gen2
  • Use cases of Data Lake Storage Gen2
  • Processing big data using Data Lake Store
  • Uploading data to Data Lake Storage
  •  Security & Authentication

Data Streaming with Azure Streaming Analytics

  • Intro to Data Streams
  • Event Processing
  • Processing events with Azure Stream Analytics
  • Messaging for big data applications using Azure Event Hubs
  • Configuring applications to send and receive messages
  • Processing Streaming data with Azure Stream Analytics

Key Takeaways:

  • Practical oriented sessions
  • Real-time examples & scenarios
  • Real-time Projects & case studies
  • Clear Azure Fundamentals exam (AZ 900)
  • Clear Azure Data Engineer Associate exam (DP 203)

Hands-on Exercises in every concept


Follow us on