- Reserved Memory: 300MB reserved for Spark internal objects.
- User Memory: UDFs, user data structures.
- Unified Memory (M): Shared between Storage (cached DataFrames) and Execution (shuffle, joins).
Default split is 50% Storage / 50% Execution, but Spark dynamically rebalances as needed.
- Spark UI (port 4040): Job, stage, executor, storage, environment tabs.
- Spark History Server: Long-term storage of completed applications.
- Ganglia / Prometheus: Cluster-level metrics.
- Databricks Cluster UI: Built-in cluster metrics for Databricks users.
Key metrics to watch: executor GC time (high GC = memory pressure), shuffle spill, task duration variance, straggler tasks.
Common Architecture Pitfalls
- Too few executors: Wastes parallelism. Aim for 2–4 cores per executor.
- Too small executors: High overhead. 4–8 GB per executor minimum for production.
- Excessive shuffle: Avoid groupByKey; use reduceByKey or aggregateByKey.
- Data skew: A few very large partitions slow the entire job. Use salting or repartition.
- Driver OOM: collect() on huge datasets crashes the driver. Use write or take(N) instead.
Common Spark Architecture Pitfalls
Watch out for these traps:
- Excessive shuffle: groupByKey is expensive. Use reduceByKey or aggregateByKey which perform local aggregation before shuffling.
- Wrong executor sizing: 4–8 GB per executor with 2–4 cores is a good starting point. Too small = overhead; too big = GC pressure.
- Driver OOM: collect() on big DataFrames crashes the driver. Always write or stream results back.
- No speculative execution: Enable spark.speculation=true for long-running jobs to handle stragglers.
- Ignoring the Spark UI: The UI shows shuffle spill, GC time, and straggler tasks. Review it for every production job.
Quick Reference — Cheatsheet
- Driver = brain (schedules tasks); Executors = muscles (run tasks).
- Cluster Manager allocates executors — YARN, K8s, or Standalone.
- Stages split at shuffle boundaries — wide transformations create new stages.
- Tasks = one partition of data per executor.
- Watch executor GC time and shuffle spill — the two main production bottlenecks.
Spark Architecture Quick-Wins
Apply these patterns for production Spark jobs:
- Set spark.sql.shuffle.partitions explicitly: Default is 200. Tune to your data volume — 2–4x total cores is a good starting point.
- Enable dynamic allocation: spark.dynamicAllocation.enabled=true scales executors up/down based on workload.
- Use broadcast joins for small tables: Spark automatically broadcasts <10MB tables. For larger, use broadcast(df) explicitly.
- Tune executor memory: spark.executor.memory=8g, spark.executor.cores=4 is a good starting point. Increase for memory-heavy workloads.
- Monitor with the Spark History Server: Configure the History Server to retain logs for 30+ days so you can debug past jobs.
Frequently Asked Questions
What is a Driver in Spark?
The process that runs main(). Builds DAG, schedules tasks, holds SparkSession. Runs on the master node.
What is an Executor in Spark?
A JVM process on a worker node. Runs tasks and stores data in cache. Each app gets its own executors.
What is a Cluster Manager in Spark?
Allocates resources to the application. Options: Standalone, YARN, Kubernetes, Mesos.
How does Spark execute a job?
Driver builds DAG → DAG splits into stages → Task scheduler launches tasks → Executors run tasks → Results return.
What is the difference between stages and tasks?
A stage is a sequence of narrow transformations. A task is a unit of work sent to one executor (one partition).
How do I monitor Spark architecture in production?
Spark UI (4040), History Server, Ganglia, Prometheus, Databricks Cluster UI. Watch GC time, shuffle spill, task variance.
Fazlur Rahman
