Spark Architecture — Quick Answer
Spark's architecture has three main components: the Driver (runs main(), builds DAG, schedules tasks), Executors (JVM processes on workers that run tasks and cache data), and the Cluster Manager (allocates resources — YARN, K8s, or Standalone). When you submit a Spark job, the driver builds a DAG, splits it into stages, and the task scheduler launches tasks on executors across the cluster.
Spark Architecture — The Three Layers
| Component | Where It Runs | Responsibility |
|---|---|---|
| Driver | Master node | Runs main(), builds DAG, schedules tasks, holds SparkSession |
| Cluster Manager | Master node | Allocates resources to the application (Standalone, YARN, K8s, Mesos) |
| Executor | Worker nodes | Runs tasks, stores data in memory/disk cache |
The driver is the brain — it doesn't process data. Executors do all the actual computation.
Spark Cluster Manager Options
| Manager | Best For | Pros | Cons |
|---|---|---|---|
| Standalone | Small/medium clusters | Simple to set up | Limited features |
| YARN | Hadoop shops | Mature, integrates with HDFS | Complex |
| Kubernetes | Cloud-native | Container-native, scalable | Setup overhead |
| Mesos | Legacy | Multi-framework | Declining adoption |
| Databricks | Managed Spark | Photon, DLT, Unity Catalog | Vendor lock-in |
Kubernetes is the most modern choice in 2026. Databricks is the most popular managed offering.
How a Spark Job Executes End-to-End
- Driver launches: main() runs, SparkSession is created.
- DAG built: RDD/DataFrame transformations are recorded as a logical plan.
- Optimisation: Catalyst optimiser rewrites the plan for efficiency.
- Stages split: DAG scheduler splits the DAG into stages at shuffle boundaries.
- Tasks launched: Task scheduler sends tasks to executors.
- Executors run tasks: Each task processes one partition of data.
- Results aggregate: Output returns to the driver (or is written to storage).
Stages vs Tasks
| Concept | Definition | When It Happens |
|---|---|---|
| Job | Triggered by an action (count, write, collect) | Each action creates one job |
| Stage | Sequence of narrow transformations | Wide transformations create new stages |
| Task | Unit of work sent to one executor (one partition) | Each task processes ~100MB–1GB of data |
Executor Memory Layout
An executor's memory is divided into:
- Reserved Memory: 300MB reserved for Spark internal objects.
- User Memory: UDFs, user data structures.
- Unified Memory (M): Shared between Storage (cached DataFrames) and Execution (shuffle, joins).
Default split is 50% Storage / 50% Execution, but Spark dynamically rebalances as needed.
Monitoring Tools
- Spark UI (port 4040): Job, stage, executor, storage, environment tabs.
- Spark History Server: Long-term storage of completed applications.
- Ganglia / Prometheus: Cluster-level metrics.
- Databricks Cluster UI: Built-in cluster metrics for Databricks users.
Key metrics to watch: executor GC time (high GC = memory pressure), shuffle spill, task duration variance, straggler tasks.
Common Architecture Pitfalls
- Too few executors: Wastes parallelism. Aim for 2–4 cores per executor.
- Too small executors: High overhead. 4–8 GB per executor minimum for production.
- Excessive shuffle: Avoid groupByKey; use reduceByKey or aggregateByKey.
- Data skew: A few very large partitions slow the entire job. Use salting or repartition.
- Driver OOM: collect() on huge datasets crashes the driver. Use write or take(N) instead.
Common Spark Architecture Pitfalls
Watch out for these traps:
- Excessive shuffle: groupByKey is expensive. Use reduceByKey or aggregateByKey which perform local aggregation before shuffling.
- Wrong executor sizing: 4–8 GB per executor with 2–4 cores is a good starting point. Too small = overhead; too big = GC pressure.
- Driver OOM: collect() on big DataFrames crashes the driver. Always write or stream results back.
- No speculative execution: Enable spark.speculation=true for long-running jobs to handle stragglers.
- Ignoring the Spark UI: The UI shows shuffle spill, GC time, and straggler tasks. Review it for every production job.
Quick Reference — Cheatsheet
- Driver = brain (schedules tasks); Executors = muscles (run tasks).
- Cluster Manager allocates executors — YARN, K8s, or Standalone.
- Stages split at shuffle boundaries — wide transformations create new stages.
- Tasks = one partition of data per executor.
- Watch executor GC time and shuffle spill — the two main production bottlenecks.
Spark Architecture Quick-Wins
Apply these patterns for production Spark jobs:
- Set spark.sql.shuffle.partitions explicitly: Default is 200. Tune to your data volume — 2–4x total cores is a good starting point.
- Enable dynamic allocation: spark.dynamicAllocation.enabled=true scales executors up/down based on workload.
- Use broadcast joins for small tables: Spark automatically broadcasts <10MB tables. For larger, use broadcast(df) explicitly.
- Tune executor memory: spark.executor.memory=8g, spark.executor.cores=4 is a good starting point. Increase for memory-heavy workloads.
- Monitor with the Spark History Server: Configure the History Server to retain logs for 30+ days so you can debug past jobs.
Frequently Asked Questions
What is a Driver in Spark?
The process that runs main(). Builds DAG, schedules tasks, holds SparkSession. Runs on the master node.
What is an Executor in Spark?
A JVM process on a worker node. Runs tasks and stores data in cache. Each app gets its own executors.
What is a Cluster Manager in Spark?
Allocates resources to the application. Options: Standalone, YARN, Kubernetes, Mesos.
How does Spark execute a job?
Driver builds DAG → DAG splits into stages → Task scheduler launches tasks → Executors run tasks → Results return.
What is the difference between stages and tasks?
A stage is a sequence of narrow transformations. A task is a unit of work sent to one executor (one partition).
How do I monitor Spark architecture in production?
Spark UI (4040), History Server, Ganglia, Prometheus, Databricks Cluster UI. Watch GC time, shuffle spill, task variance.
