8 Apache Spark Interview Questions: Plans, Skew, Caching, and Correctness
Quick Overview
Prepare for Spark interviews with evidence-based answers on execution, shuffles, caching, join skew, partition counts, and output layout.
When a Spark job becomes slower, start with evidence: input volume, the executed plan, task durations, shuffle traffic, spill, and executor failures. Naming repartition() or adding memory is not a diagnosis. The useful answer explains which observation supports a change and which result would make you abandon it.
These eight questions cover that reasoning. They describe batch Spark workloads; storage defaults and optimizer behavior depend on the API and deployed version. Use the public Spark execution and optimization question to rehearse the explanation, then check the assumptions against your own runtime.
1. How do RDDs, DataFrames, and Datasets differ?
An RDD exposes partitioned data and application functions. A DataFrame exposes named columns and a schema, giving Spark SQL more information for relational optimization. Scala and Java also have typed Dataset APIs; PySpark does not provide the same typed Dataset interface.
For structured event data, begin with an explicit schema, column selection, filters, and built-in expressions. An opaque application function can hide opportunities that a column expression exposes. This does not mean every built-in expression is cheaper or that every Python UDF has the same execution path: ordinary scalar UDFs, Arrow-based execution, and vectorized UDFs differ.
Use an RDD when its lower-level control fits the problem, and explain what you give up. The RDD programming guide documents transformations, shuffle operations, and persistence. Its closures section also explains why mutating a driver-side variable inside foreach is undefined behavior: tasks on separate executors update serialized copies, so the driver's value can remain unchanged. Use an accumulator for that kind of aggregation; a driver-side variable and a distributed dataset have different execution boundaries.
Follow-up: inspect the logical and physical plan. Identify scans, expressions, exchanges, and Python execution operators rather than claiming an optimization from the API name alone.
2. What does lazy evaluation mean?
Transformations describe work; an action requests a result or side effect. Defining a filter can be quick because it has not yet scanned and processed the input. count(), collect(), and an executed write trigger computation, but only collect() tries to return the entire result to the driver.
Several actions over an expensive uncached intermediate can repeat upstream work. Persistence may help when reuse saves more computation than materialization, storage, and eviction cost. It is not automatically useful for an intermediate consumed once.
Check the Jobs and SQL tabs for repeated scans and stages. A single user action is not a promise of exactly one Spark job: query planning and adaptive execution can introduce additional execution work. For broader troubleshooting, use the Spark shuffles and debugging article.
Follow-up: a small limit or sample is a bounded inspection tool, not proof that the whole dataset fits in driver memory.
3. Why do shuffles create important boundaries?
A narrow dependency lets downstream work use particular parent partitions without a broad redistribution. A shuffle redistributes records so matching keys can meet downstream. Network transfer, serialization, disk I/O, and spill can make that movement expensive, but a necessary redistribution should not be removed at the expense of correctness.
For associative and commutative reductions on keyed RDD data, such as reduceByKey with a sum, local combining can reduce shuffle traffic compared with collecting every value for a key. The reduction must obey the required algebraic rules; arbitrary order-sensitive functions are not interchangeable with a sum.
Not every join requires the same exchanges. A broadcast strategy or compatible existing distribution can avoid particular redistribution steps. Inspect the actual plan and stage metrics. A method called join is less informative than its selected physical strategy.
Practice this distinction with the public MapReduce and Spark job-design prompt and the RDD aggregation article.
4. What runs on the driver and executors?
The driver coordinates the application and schedules tasks. Executors run tasks and retain application data such as cached partitions. A cluster manager allocates resources; it does not replace the driver's task scheduler. Spark's current cluster overview describes standalone, YARN, and Kubernetes deployments.
Jobs contain stages, and stages contain tasks operating on partitions. Shuffle dependencies help explain stage boundaries. Driver and executor failures require different investigations: more executors will not repair driver memory exhaustion caused by collecting a large result.
In client mode, the submitting process hosts the driver and must remain reachable. Cluster mode places the driver within the managed cluster environment. Choose based on the deployment contract and connectivity requirements, not a claim that one mode makes every query faster.
Follow-up: identify where a failure appears, then separate a lost executor, a failed task, a stalled external dependency, and an application-level exception.
5. When should you cache or persist data?
cache() selects the API's default persistence level; persist() allows an explicit level. Do not apply an RDD default to a DataFrame without checking. The PySpark DataFrame.persist API specifies MEMORY_AND_DISK_DESER as its default, while RDD caching uses MEMORY_ONLY by default. Representations also differ between JVM and Python data.
Persistence is lazy. An action stores the partitions it computes; a limited action need not populate the entire cache. Inspect the Storage tab and later execution plans before saying the result was fully materialized or reused. An existing persistence level also needs deliberate handling before changing it; calling persist again is not a general storage-level replacement procedure.
Cache an expensive shared intermediate after unnecessary data has been removed. Unpersist it when reuse ends. Cached data can compete with execution work for resources, and lost or evicted partitions may need recomputation. Persistence is not a durable dataset publication or backup mechanism.
6. How do you diagnose skew and choose a join strategy?
If most tasks finish but a few continue with much larger inputs, investigate uneven key frequencies and partition sizes. Compare maximum and typical task durations, shuffle reads, spill, and the executed join plan. A large cluster cannot automatically divide every hot-key computation.
A broadcast join can suit a sufficiently small build relation, but compressed file size is not a complete estimate of its in-memory cost. Check statistics, executor memory, concurrent tasks, and join semantics. A broadcast hint cannot manufacture memory or make an unsupported strategy valid.
Adaptive Query Execution can change eligible plans using runtime statistics, including coalescing shuffle partitions and addressing supported skewed joins. It does not repair every skew pattern. Explain the initial and final plans using the AQE practice article and Spark's SQL tuning reference.
Salting needs a correctness argument: which side receives salts, which matching rows are replicated, and how is join multiplicity preserved? Extra keys without that explanation can produce a different answer.
7. How do repartition and coalesce affect parallelism?
repartition redistributes data and can increase or decrease partition counts. Key-based repartitioning can still be skewed when one key dominates. coalesce usually reduces partitions through a narrow dependency; it does not evenly redistribute individual records, and asking for more partitions does not make it an increasing-partition operation.
A drastic reduction can reduce upstream parallelism as well as the number of output tasks. The coalesce API documents this consequence. Prefer evidence about task sizes, scheduling overhead, and available resources to a universal partition-size rule.
| Observation | Candidate investigation | Limit to check |
|---|---|---|
| Few long tasks, idle capacity | Distribution and partition count | A hot key may remain indivisible |
| Many very short tasks | Scheduling overhead and small inputs | Fewer tasks can reduce useful parallelism |
| Spill and oversized tasks | Data reduction, distribution, memory use | More memory may hide a bad plan |
Change one major factor at a time when possible, keeping input and resource conditions comparable. These are diagnostic hypotheses, not measured speedup claims.
8. How do you address small output files?
Separate execution partitions from directory partition columns. Writing by a high-cardinality identifier can create many directories and small files even if the task count initially looks reasonable. Conversely, Spark can group multiple small input files into scan partitions, so one file does not necessarily imply one read task.
Choose a format and layout for representative downstream filters. Estimate output parallelism, inspect actual object counts and size distributions, and compare later scan behavior. coalesce(1) is suitable for some tiny exports; it is a bottleneck for a large analytical dataset.
Periodic compaction may help frequently appended tables, but the supported command and transaction guarantees belong to the table format and deployed platform. It adds maintenance work and should be assessed alongside read cost. A directory-partitioned write does not guarantee exactly one file per execution partition.
Validate correctness before accepting a faster plan
Use a tiny original join fixture before a large run: facts (A,5), (A,7), (B,11), (NULL,13) and dimension keys A, A, C. An ordinary equality left join produces six rows and a naive sum of 48, although the fact total is 36. The repeated dimension key duplicates both A facts; unmatched B and NULL facts remain once.
Decide whether repeated dimension keys are valid, whether a version predicate is missing, or whether the join contract requires rejecting duplicates. Do not silently deduplicate a dimension if its rows represent meaningful history. This fixture checks relational semantics; it is not a Spark performance benchmark.
Then rehearse the public pipeline reliability, backfills, and Spark optimization question. State the suspected bottleneck, evidence, proposed change, downside, and correctness check. A useful diagnosis remains defensible when the data distribution changes.
Comments (0)