Applied Data Pipelines using Azure Data Factory and Microsoft Fabric

Course Outline

Azure Data Factory Overview

• Understanding ETL
• Understanding Data Pipeline and Data Flow
• Evolution of data integration from on-premises to cloud-based solutions
• Understanding Azure Data Factory
• Key concepts in Azure Data Factory: pipelines, activities, datasets, triggers

Building Data Pipelines in Azure Data Factory

• Introduction to Data Movement activities
• Using Copy Data activity
• Using Data Flow activity
• Using Custom activity
• Introduction to Data Transformation activities
• Using Mapping Data Flow
• Using Wrangling Data Flow
• Using Stored Procedure activity
• Working with Variables
• Working with Lookup
• Working with Wait

Integration with External Services

• Integration with Azure services: Azure Blob Storage, Azure SQL, and Azure Synapse Analytics
• Integration with external services: Amazon S3, Google Cloud Storage, Salesforce

Data Transformation Techniques

• Working with Multiple inputs/outputs
• Working with Schema modifier
• Working with Formatters
• Mapping Data Flow
• Wrangling Data Flow
• Implementing data cleansing, enrichment, and aggregation

Data Orchestration and Workflow Management

Managing dependencies and scheduling in Azure Data Factory

Implementing parameterized pipelines for dynamic data processing

Configuring triggers

Working with parameters

Using dynamic content in data pipelines

Working with Iteration & Conditionals

• Using ForEach and Until
• Using Filter
• Using If Condition & Switch

Monitoring and Management in Azure Data Factory

• Monitoring data pipelines using Azure Data Factory Monitoring Hub
• Understanding pipeline run status, triggers, and alerts
• Managing data factory resources, including linked services, datasets, and pipelines
• Implementing best practices for performance optimization and cost management

Synapse Pipelines Integration

• Leveraging Azure Synapse Analytics for scalable data processing
• Integrating Synapse pipelines with Azure Data Factory for end-to-end data workflows
• Understanding Synapse Pipelines and their role in data processing
• Configuring Data Movement activities between Azure Data Factory and Azure Synapse Analytics
• Utilizing Synapse Linked Services and Datasets in Azure Data Factory pipelines

Pipelines in Fabric

• Understanding and implementing Pipelines in Fabric for large-scale data processing
• Leveraging Data Flows in Pipelines in Fabric for complex data transformations
• Creating and managing Data Marts within Pipelines in Fabric for optimized data storage and retrieval
• Utilizing Datasets in Pipelines in Fabric for defining data structures and schemas
• Implementing best practices for designing and orchestrating Pipelines in Fabric
• Hands-on exercises: Building and optimizing data pipelines using Pipelines in Fabric

Best Practices in Azure Data Factory

• Performance tuning for data pipelines
• Parallel execution and partitioning strategies for improved throughput
• Data skew management and load balancing techniques
• Pipeline design best practices: modularization, reusability, and maintainability
• Error handling and exception management strategies
• Version control and deployment best practices
• Resource optimization
• Leveraging serverless compute options and auto-scaling capabilities.