Microsoft Certification: Azure Data Engineer Associate (DP-203)
Skills measured in DP-203:
- Design and implement data storage solutions
- Design and develop data processing (building and deploying data pipelines)
- Design and implement data security
- Monitor and optimize data storage and data processing
Key Skills covered:
| Azure Blob Storage | Azure Cosmos DB | Azure SQL | Azure Data Lake Storage |
| Azure Data Factory | Azure Synapse Analytics | Azure Databricks | Azure Stream Analytics |
| Azure Batch | Azure Pipelines | Azure Monitor | Azure HDInsight |
Azure for the Data Engineer
- Cloud Computing basics
- Introduction to Microsoft Azure
- Applications of Azure
- Azure Services
- Understanding Data
- On-premises vs Cloud-based servers
- Azure Data Engineer roles & tasks
- Data Engineering processes
- Use cases for the Cloud
Data Services on Azure Data Platform
- Data types & categories
- Understanding Azure Data storage
- Data storage in Azure Data Lake Storage
- Azure Cosmos DB
- Azure SQL database
- Azure Synapse Analytics
- Azure Stream Analytics
- Azure HDInsight
- Azure Databricks
- Azure Data Factory
Data Storage in Azure
Non-Relational data stores
- Document data stores
- Columnar data stores
- Key/value data stores
- Graph data stores
- Object data stores
Azure Blob storage
Azure Cosmos DB
- Data Partitioning: Horizontal/Vertical/Functional
- Consistency levels in Cosmos DB
Non-Relational Database vs NoSQL
Azure Data Lake Storage
- Why Data Lake?
- Data Lake architecture
- Data Lake key components
- How data lake stores data
- Azure Data Lake Storage Gen2
Azure SQL Database
- Deployment models
- High-Availability models
Azure Database for MySQL
PolyBase
Azure Synapse Analytics (Azure SQL Datawarehouse)
- SQL Analytics and SQL Pool in Azure Synapse
- Key components of big data solution
- MPP architecture
Data Integration with Azure Data Factory (ADF)
- Data integration patterns
- ADF components
- Linked services
- Datasets
- Activities and Pipelines
- Integration runtimes
- ADF ingestion methods
- ADF connectors
- Copy activity
- ADF Transformation methods
- ADF Mapping data flow
- ADF Control flow
- Working with ADF pipelines
- Executing data factory packages
- Monitor ADF pipelines
Data Warehousing Solutions with Azure Synapse Analytics
- Intro to Azure Synapse Analytics (Azure SQL Datawarehouse)
- How Azure Synapse Analytics works
- When to use Azure Synapse Analytics
- Create Azure Synapse Analytics workspace
- Synapse Analytics SQL
- Creating Apache Spark Pools
- Apache Spark Notebooks
- Load data in Spark notebooks
- Load data in Spark DataFrames
- Integrating SQL and Spark Pools
- Orchestrating data integrating with Synapse Analytics Pipelines
- Understanding Synapse Analytics processes
- Working with Azure Synapse Studio
- Data hub
- Develop hub
- Integrate hub
- Monitor & Manage hub
- Designing modern data warehouse architecture components
- Designing Ingestion patterns
- Data storage
- Transforming data with Synapse Analytics
- Monitoring data warehouse activities in Synapse Analytics
- Column and row-level security
Data Engineering with Azure Databricks
- Understanding Azure Databricks
- Databricks Notebooks
- Working with Databricks Notebooks
- Databricks Spark Cluster
- Architecture of Spark job
- Read & Write data in Azure Databricks
- Working with DataFrames in Azure Databricks
- DataFrames Transformations and Functions
- Eager and Lazy execution
- Security & Data protection in Azure Databricks
- Delta Lakes to create, append data to Apache Spark tables
- Structured streaming
- Scheduling Databricks jobs in a data factory pipeline
Large-Scale Data Processing with Azure Data Lake Storage
- Intro to Data Lake Storage
- Understanding Data Lake Storage Gen2
- Use cases of Data Lake Storage Gen2
- Processing big data using Data Lake Store
- Uploading data to Data Lake Storage
- Security & Authentication
Data Streaming with Azure Streaming Analytics
- Intro to Data Streams
- Event Processing
- Processing events with Azure Stream Analytics
- Messaging for big data applications using Azure Event Hubs
- Configuring applications to send and receive messages
- Processing Streaming data with Azure Stream Analytics
Key Takeaways:
- Practical oriented sessions
- Real-time examples & scenarios
- Real-time Projects & case studies
- Clear Azure Fundamentals exam (AZ 900)
- Clear Azure Data Engineer Associate exam (DP 203)
Hands-on Exercises in every concept