Fazlur Rahman
Apache Airflow Explained — Quick Answer
Apache Airflow is an open-source workflow orchestration platform. You define workflows as Directed Acyclic Graphs (DAGs) in code, and Airflow schedules, monitors, and manages them. It is the de facto standard for batch data pipeline orchestration in 2026 — used by Airbnb, Google, Amazon, Uber, and most modern data teams. Alternatives include Prefect and Dagster, but Airflow remains the most widely adopted.
Core Airflow Concepts
| Concept | Description |
|---|---|
| DAG | Directed Acyclic Graph — a Python file defining the workflow |
| Task | A unit of work in a DAG |
| Operator | Template for a task (BashOperator, PythonOperator, etc.) |
| Sensor | A special operator that waits for an external condition (file, SQL row, API) |
| Scheduler | The Airflow component that triggers tasks at the right time |
| Executor | How tasks are run (Sequential, Local, Celery, Kubernetes) |
| Metadata DB | Stores DAG runs, task states, XComs (PostgreSQL/MySQL) |
Sample DAG
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from datetime import datetime
with def:
extract = BashOperator(
task_id='extract',
bash_command='python /scripts/extract.py',
)
transform = PythonOperator(
task_id='transform',
python_callable=transform_func,
)
load = BashOperator(
task_id='load',
bash_command='python /scripts/load.py',
)
extract >> transform >> loadAirflow Executors Compared
| Executor | How It Runs Tasks | Best For |
|---|---|---|
| SequentialExecutor | One task at a time, in the scheduler process | Dev/testing only |
| LocalExecutor | Parallel within the scheduler process | Small workloads, single-machine |
| CeleryExecutor | Distributed via Celery + Redis/RabbitMQ | Production, fixed cluster size |
| KubernetesExecutor | Each task runs in its own K8s pod | Production, dynamic scaling, isolation |
| CeleryKubernetesExecutor | Mix of Celery + K8s | Hybrid workloads |
KubernetesExecutor is increasingly the standard — each task runs in an isolated pod, with full dependency isolation.
