Skip to content

Conversation

@SakshamSinghal20
Copy link
Contributor

@SakshamSinghal20 SakshamSinghal20 commented Feb 11, 2026

closes: #61700

Description

Deferred tasks were previously excluded from max_active_tis_per_dag and max_active_tis_per_dagrun concurrency checks. This allowed an unlimited number of tasks to enter the DEFERRED state, bypassing configured limits.

This PR enforces these limits for DEFERRED task instances while ensuring that max_active_tasks (which limits worker slot usage) remains unaffected.

Changes:

  • airflow/ti_deps/dependencies_states.py: Introduced TASK_CONCURRENCY_EXECUTION_STATES to include DEFERRED, RUNNING, and QUEUED.
  • airflow/jobs/scheduler_job_runner.py: Updated ConcurrencyMap to count deferred TIs for task-level limits, but explicitly exclude them from dag_run_active_tasks_map.
  • airflow/models/taskinstance.py: Updated get_num_running_task_instances to count deferred TIs for pre-execution dependency checks.

Tests:

Added unit tests covering:

  • Deferred tasks blocking new scheduling attempts.
  • Mixed state scenarios (Running + Deferred).
  • Regression checks ensuring max_active_tasks is not impacted.

Was generative AI tooling used to co-author this PR?
  • Yes (used for guidance and file locating)

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

max_active_tis_per_dag not enforced for deferred task instances

1 participant