Apache Airflow: Workflow Orchestration (2026)

Apache Airflow Explained — Quick Answer

Apache Airflow is an open-source workflow orchestration platform. You define workflows as Directed Acyclic Graphs (DAGs) in code, and Airflow schedules, monitors, and manages them. It is the de facto standard for batch data pipeline orchestration in 2026 — used by Airbnb, Google, Amazon, Uber, and most modern data teams. Alternatives include Prefect and Dagster, but Airflow remains the most widely adopted.

Core Airflow Concepts

Sample DAG

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from datetime import datetime

with def:
    extract = BashOperator(
        task_id='extract',
        bash_command='python /scripts/extract.py',
    )

    transform = PythonOperator(
        task_id='transform',
        python_callable=transform_func,
    )

    load = BashOperator(
        task_id='load',
        bash_command='python /scripts/load.py',
    )

    extract >> transform >> load

Airflow Executors Compared

KubernetesExecutor is increasingly the standard — each task runs in an isolated pod, with full dependency isolation.

Airflow vs Prefect vs Dagster

Airflow wins on ecosystem and maturity. Prefect and Dagster win on modern developer experience.

Top Airflow Use Cases

• ETL/ELT pipelines: Schedule Spark, dbt, warehouse loads.
• ML pipelines: Schedule training, evaluation, deployment.
• Data quality checks: Trigger Soda or Great Expectations tests.
• Reporting: Generate daily/weekly reports from warehouses.
• Cross-system orchestration: Coordinate AWS, GCP, Azure, and SaaS jobs.

Airflow Best Practices

1. Use TaskFlow API (Airflow 2.0+): Cleaner decorator-based DAGs with @task.
2. Set retries: retries=3, retry_delay=timedelta(minutes=5) on every task.
3. Use SLA and alerting: Catch missed or slow tasks.
4. Parameterise with Variables and Connections. Don't hardcode.
5. Use the Astro CLI or managed services (Astronomer, MWAA, GCP Cloud Composer) for production.
6. Test DAGs with pytest. Mock external dependencies.

Common Airflow

Real-World Example — A Production Airflow DAG

Here is a realistic ETL DAG that runs every night:

1. 00:00 — Ingest: SparkSubmitOperator runs a Spark job to ingest raw events from S3 into the warehouse.
2. 02:00 — dbt run: BashOperator runs "dbt run --select staging+ marts+" to refresh staging and mart models.
3. 03:00 — dbt test: BashOperator runs "dbt test" — fails the DAG if any data quality test fails.
4. 03:30 — Soda checks: BashOperator runs Soda scans for additional quality validation.
5. 04:00 — Refresh BI: HttpOperator triggers Looker/Power BI refresh via API.
6. 04:30 — Notify: SlackOperator posts a success message to the data engineering channel.

This DAG runs on KubernetesExecutor — each task runs in an isolated pod with the right dependencies.

Pitfalls

Avoid these traps when building Airflow DAGs:

• Hardcoding values in DAGs: Use Variables and Connections for any value that might change.
• No retries: Every task should have retries=3 with a meaningful retry_delay. Transient failures are normal.
• Long-running tasks: Tasks running more than 1 hour should be split or moved to Spark. Airflow's scheduler has timeout limits.
• No testing: Use pytest + airflow.models.DagBag to test DAGs in CI. Broken DAGs should fail the build.
• LocalExecutor in production: LocalExecutor runs tasks in the scheduler process. Use CeleryExecutor or KubernetesExecutor for production.

Quick Reference — Cheatsheet

• DAGs are Python files — no cycles allowed.
• Use TaskFlow API (Airflow 2.0+) for cleaner decorator-based DAGs.
• Set retries=3, retry_delay=5min on every task.
• KubernetesExecutor is the modern choice — each task = isolated pod.
• Always use managed Airflow (Astronomer, MWAA, Composer) for production.

Airflow Quick-Wins

Apply these patterns for production Airflow deployments:

• Use Pools for resource constraints: If multiple tasks hit the same API or database, use a Pool to limit concurrency.
• Set SLA on every DAG: sla_miss_callback alerts the team if a task exceeds the expected duration.
• Use XCom sparingly: XCom is fine for small metadata but not for large data. Use external storage (S3) for large payloads.
• Schedule with cron expressions: "0 2 * * *" runs at 2 AM daily. Use the Airflow UI to test cron strings.
• Enable task-level logging: Set AIRFLOW__LOGGING__REMOTE_LOGGING=True to ship logs to S3/GCS for long-term retention.

Frequently Asked Questions

What is Apache Airflow?

An open-source workflow orchestration platform. Workflows are defined as Python DAGs.

What is a DAG in Airflow?

A Python file defining a workflow — tasks + dependencies. No cycles allowed.

What is an Operator in Airflow?

A template for a task. BashOperator (shell), PythonOperator (Python), PostgresOperator (SQL), SparkSubmitOperator (Spark).

What Airflow executors are available?

Sequential, Local, Celery, Kubernetes, CeleryKubernetes. Production uses Celery or Kubernetes.

Is Airflow a streaming engine?

No — Airflow is batch. For streaming, use Flink, Spark Streaming, or Kafka Streams.

How does Airflow compare to Prefect or Dagster?

Airflow: most adopted, largest ecosystem. Prefect: cleaner developer experience. Dagster: software-defined assets.

Photo: Jernej Furman from Slovenia, CC BY, via Wikimedia Commons (https://upload.wikimedia.org/wikipedia/commons/5/5a/AI_Artificial_Intelligence_concept_%2852917075159%29.jpg?utm_source=commons.wikimedia.org&utm_campaign=imageinfo&utm_content=original)