Airflow control the parallelism and concurrency (draw)
Airflow configuration to allow for a larger scheduling capacity and frequency:
DAGs have configurations that improve efficiency:
max_active_tasks
: Overrides max_active_tasks_per_dag.max_active_runs
: Overrides max_active_runs_per_dag.
Operators or tasks also have configurations that improves efficiency and scheduling priority:
max_active_tis_per_dag
: This parameter controls the number of concurrent running task instances acrossdag_runs
per task.pool
: See Pools.priority_weight
: See Priority Weights.queue
: See Queues for CeleryExecutor deployments only.