- Narrow transformations: map, filter — can be pipelined in one stage.
- Wide transformations: groupBy, join — require shuffle across the cluster.
Where Spark Runs
- Standalone: Spark's own cluster manager — small/medium.
- YARN: Hadoop YARN — common in on-prem Hadoop deployments.
- Kubernetes: Modern, container-native — most common in 2026.
- Apache Mesos: Legacy — rarely used now.
- Managed cloud: Databricks, AWS EMR, Azure HDInsight, GCP Dataproc.
Spark vs Hadoop MapReduce
| Aspect | MapReduce | Spark |
|---|
| Speed | Disk-based, slow | In-memory, 10–100x faster |
| Ease of Use | Low (verbose Java) | High (Python/Scala APIs) |
| Streaming | Micro-batch (slow) | Structured Streaming (real-time) |
| ML | Mahout (limited) | MLlib (rich) |
| Adoption | Legacy | Standard |
Common Spark Real-World Example — Building Your First Spark Job
Walk through building a simple Spark job step by step:
- Setup: Install PySpark (pip install pyspark) and create a local SparkSession.
- Read data: spark.read.csv("data.csv", header=True, inferSchema=True).
- Transform: Filter, group, aggregate using DataFrame API.
- Write output: result.write.parquet("output/") saves the result to Parquet format.
- Submit to cluster: spark-submit --master yarn --deploy-mode cluster job.py on a YARN cluster.
This pattern — read, transform, write — is the foundation of every Spark job. From here, you can scale to billions of rows with the same API.
Pitfalls for Beginners
Avoid these traps when starting with Spark:
- Using RDDs when DataFrames would work: DataFrames get Catalyst optimisation. RDDs don't. Default to DataFrames for almost every use case.
- Calling collect() on large DataFrames: collect() pulls all data to the driver — can OOM on big data. Use write or take(N) instead.
- Ignoring data skew: A few very large partitions slow down the entire job. Use salting or repartition.
- Too few or too many partitions: Aim for 2–4x the total cores in the cluster. Too few wastes parallelism; too many adds overhead.
- Forgetting to broadcast small DataFrames: broadcast() small lookup tables (<10MB) to avoid expensive shuffle joins.
Quick Reference — Cheatsheet
- Use DataFrames, not RDDs — Catalyst optimiser gives 5–10x speedup.
- PySpark is the most popular API — Python for data engineers.
- Lazy evaluation: transformations build DAG; actions trigger execution.
- Wide transformations (groupBy, join) trigger shuffles — expensive.
- Kubernetes is the modern cluster manager; Databricks is the most popular managed service.
Spark Quick-Wins for Beginners
Once you have the basics, apply these Spark patterns to write efficient code from day one:
- Use persist() for repeated DataFrames: If you reuse a DataFrame in multiple actions, call persist() to cache it in memory.
- Avoid UDFs when built-in functions work: UDFs break Catalyst optimisation. Use built-in functions (pyspark.sql.functions) wherever possible.
- Use coalesce() to reduce partition count: Use coalesce(N) — not repartition(N) — to combine partitions without a shuffle.
- Predicate pushdown: Spark pushes WHERE clauses down to Parquet/ORC readers. Filter early, on partitioned columns where possible.
- Watch the DAG in the Spark UI: Every time you write a query, look at the DAG and stages. Shuffles = expensive stages.
Frequently Asked Questions
What is Apache Spark?
An open-source distributed data processing engine — 100x faster than MapReduce via in-memory computation. Supports SQL, DataFrames, MLlib, Streaming.
What is an RDD in Spark?
Resilient Distributed Dataset — Spark's fundamental data structure. Immutable, distributed collection. Supports transformations (lazy) and actions.
What is the difference between RDD and DataFrame?
RDD is low-level, type-safe. DataFrame is high-level with named columns. DataFrames get Catalyst optimisation. Most code uses DataFrames.
What languages does Spark support?
Scala (native), Python (PySpark), Java, R (SparkR). PySpark is the most popular.
Where is Spark used in production?
On YARN, Kubernetes, Mesos, standalone, or as a managed service (Databricks, EMR, HDInsight, Dataproc).
Is Spark still relevant in 2026?
Yes — Spark is the de facto engine for large-scale batch + streaming. Lakehouse platforms are built on Spark.
Fazlur Rahman
