Azure Data Factory (ADF), a powerful platform for data integration and ETL workflows, has been a go-to choice for data analytics projects. As pipeline developers, Microsoft create exciting and interesting data pipelines to meet various business needs. However, there has always been a limitation: the maximum number of activities allowed in a single pipeline. Historically, this limit stood at 40 activities per pipeline, a measure put in place to prevent resource exhaustion.
But good news! This week, Microsoft doubled that limit. We can now define up to 80 activities per pipeline. This enhancement empowers we to build more powerful, versatile, and resilient data pipelines. Let’s dive into the details.
Why the Limit?
Before we explore the new possibilities, let’s understand why there was a limit in the first place. Data Factory aims to ensure the resiliency and reliability of data pipelines. By capping the number of activities, Microsoft prevent potential resource bottlenecks. However, Microsoft recognize that our users need flexibility to create robust pipelines that handle various scenarios.
Doubling the Ceiling
With the increased limit of 80 activities, we have more freedom to design your pipelines. Here are some key points:
- Error Handling: Microsoft encourage we to use the additional 40 activities for building robust error-handling capabilities. For example:
- Send an email notification when a Copy activity fails.
- Implement a try-catch block that gracefully handles exceptions.
- Build for resilience and retries.
- Avoid Sequential Pipelines: While we can now add more activities, avoid creating excessively long sequential pipelines. Remember that data pipelines, like any software, can encounter failures. Keep the actual steps within a pipeline to a reasonable amount.
Example Scenarios
Let’s look at a couple of scenarios where the increased activity limit comes in handy:
1. Simple Error Handling
Suppose we have a pipeline with a conditional branching. we define three activities, but realistically, only two will run during any pipeline execution. Use the additional activities to enhance error handling:
Conditional Branching in Pipeline
2. Try-Catch Block
Consider a scenario where we need to move data if it’s ready or proceed otherwise. Implement a try-catch block that attempts to run the first activity. This approach ensures graceful handling of exceptions:
Try-Catch Block
As we harness the power of data pipelines, remember that Microsoft goal is to enable you to deliver business impact. With 40 more activities at our disposal, explore creative solutions, build resilient pipelines, and make your end users’ lives easier.
