PySpark Coding Interview Questions: DataFrame Tasks with Expected Outputs

PySpark coding interview questions become easier to reason about when you state the output before choosing an API. Define what one row represents, which duplicate survives, how nulls behave, and whether order matters. Then write the DataFrame transformation and compare its result with an explicit expected DataFrame.

This guide uses one original batch dataset across four connected tasks: selecting order revisions, calculating a running amount, expanding nested tags, and enriching orders before aggregation. Every expected result below was checked locally with Spark 4.0.1, Python 3.12.14, and Java 17.0.20.1 using two local worker threads. These are correctness exercises, not cluster-performance benchmarks or reported company questions.

For related practice, use PracHub's Data Engineer questions. The question bank is a source of interview prompts; this article does not claim that PracHub supplies an online Spark runtime.

PySpark practice moves from six input records to four current orders and assertions on rows and schema.

Set up an explicit input and output contract

An output contract describes the required columns, types, what each row represents, ordering, and treatment of missing values. Here, the raw grain is an order revision. The first result must contain one current record per order, while later tasks deliberately change or preserve that grain.

Run the Python snippets in sequence in a local Spark session. Install the pinned package with pip install pyspark==4.0.1 and configure a compatible Java installation. Spark's versioned installation documentation lists Python 3.9 or later and Java 17 or later for this release. PySpark installation

from pyspark.sql import SparkSession, Window, functions as F
from pyspark.testing import assertDataFrameEqual

spark = (
    SparkSession.builder.master("local[2]")
    .appName("pyspark-interview")
    .config("spark.sql.shuffle.partitions", "2")
    .getOrCreate()
)
raw = spark.createDataFrame([
    ("o1", "u1", 10, 1, 10, 100, ["a", "b"]),
    ("o1", "u1", 10, 2, 20, 120, ["a", "b"]),
    ("o1", "u1", 10, 2, 30, 125, ["a", "b"]),
    ("o2", "u1", 10, 1, 40, 80, []),
    ("o3", "u2", 20, 1, 50, None, None),
    ("o4", None, 30, 1, 60, 50, ["a"]),
], """
order_id string, user_id string, event_min int,
revision int, ingest_id int, amount long,
tags array<string>
""")

Amounts are integer units, avoiding floating-point rounding in this exercise. Event minutes are synthetic ordering values, not parsed timestamps. Revision is the primary preference; the unique ingestion identifier breaks revision ties. These are business rules supplied by the exercise, not something Spark can infer.

The missing amount means unknown, not zero. An empty tag array means the order has no tags; a null array means its tags are unknown. The null user identifier must not become a shared customer identity. Keep those distinctions visible as you work.

Task 1: choose the current order revision deterministically

Question: Return one record per order. Prefer the highest revision and, within that revision, the highest ingestion identifier. Keep the remaining columns from that winning record.

choice = Window.partitionBy("order_id").orderBy(
    F.desc("revision"), F.desc("ingest_id")
)
latest = (
    raw.withColumn("rn", F.row_number().over(choice))
    .filter("rn = 1")
    .drop("rn")
)
latest.select("order_id", "amount").orderBy("order_id").show()

Expected projection:

order_id | amount
o1       | 125
o2       | 80
o3       | NULL
o4       | 50

The two revision-2 records for o1 disagree on amount. Ingestion identifier 30 wins over 20, giving 125. Checking only that four records remain would miss a wrong survivor with amount 120 or 100.

A tempting wrong answer is raw.dropDuplicates(["order_id"]). The official API removes duplicates using the selected comparison columns; it does not express this revision-and-ingestion preference. It therefore does not satisfy the stated survivor contract. dropDuplicates documentation

Likewise, sorting a DataFrame before a key-based deduplication does not make the survivor rule explicit in that operation. Put the ordering inside the window used to choose the row.

For a follow-up, remove the uniqueness assumption on ingestion identifiers. If two otherwise tied records disagree, the contract is incomplete: request another stable preference or reject the conflict. Do not add an arbitrary tie-breaker that silently changes the business meaning.

Task 2: calculate a running amount with a defined frame

Question: For known users only, return one row per current order with the running amount ordered by event minute and then order identifier. Preserve an unknown total when a user's frame contains no known amount.

running_frame = (
    Window.partitionBy("user_id")
    .orderBy("event_min", "order_id")
    .rowsBetween(Window.unboundedPreceding, Window.currentRow)
)
running = (
    latest.filter(F.col("user_id").isNotNull())
    .withColumn("running_amount", F.sum("amount").over(running_frame))
    .select("user_id", "order_id", "running_amount")
    .orderBy("user_id", "order_id")
)
running.show()

Expected output:

user_id | order_id | running_amount
u1      | o1       | 125
u1      | o2       | 205
u2      | o3       | NULL

Both u1 orders have event minute 10. The order identifier supplies their stable within-minute sequence. The explicit row frame advances through that sequence one record at a time. Spark documents rowsBetween boundaries as inclusive; the frame above starts at the partition's beginning and ends at the current row. rowsBetween documentation

Now compare a different expression: F.sum("amount").over(Window.partitionBy("user_id").orderBy("event_min")). In the tested runtime, its default range frame includes both peers at minute 10, so both u1 rows receive 205. That is a different calculation, not a broken sum.

Two ordered rows produce cumulative amounts 125 and 205 with ROWS, while a range frame on their shared minute produces 205 for both.

Explain which question the interviewer asked. “Cumulative through this ordering value” can legitimately include all peers; “advance through individual orders” needs a stable row sequence. Choosing a frame without defining that distinction can produce convincing but incorrect output.

The null-user record is excluded intentionally. Partitioning all unknown user identifiers together would not establish that those orders belong to the same person. State this exclusion instead of allowing an accidental anonymous-user total.

Task 3: expand nested tags without losing parent orders

Question: Count non-null tag elements per current order, preserve orders with empty or null arrays, and retain a flag distinguishing unknown arrays from known-empty arrays.

flat = latest.select(
    "order_id",
    F.col("tags").isNull().alias("tags_unknown"),
    F.explode_outer("tags").alias("tag"),
)
tag_counts = (
    flat.groupBy("order_id", "tags_unknown")
    .agg(F.count("tag").alias("tag_count"))
    .orderBy("order_id")
)
tag_counts.show()

Expected output:

order_id | tags_unknown | tag_count
o1       | false        | 2
o2       | false        | 0
o3       | true         | 0
o4       | false        | 1

Spark's explode_outer emits a null element for an empty or null input array. That keeps o2 and o3 represented in the expanded data. The flag is computed from the original array, before those two cases become visually similar. explode_outer documentation

The expanded DataFrame has five rows: two for o1 and one for each other order. Ordinary explode produces only three rows here, losing the empty and unknown parents.

Do not use count("*") as the tag count after outer expansion. It counts the placeholder row for each preserved parent. Counting the tag column counts non-null elements, so both preserved parents receive zero while the flag retains their different meanings.

The result describes element counts, not distinct tag counts. If an array contains two copies of "a", this expression counts two. If the prompt requests unique tags, change the contract and transformation deliberately. Also decide whether a null element inside a non-null array indicates malformed data; the current task counts only known elements.

Task 4: enrich orders and aggregate without inventing values

Question: Attach regions using an ordinary left join on user identifier. Return order count, known-amount count, and total amount per region. Unknown users must remain unmatched, even if the dimension contains a null identifier.

users = spark.createDataFrame([
    ("u1", "North"),
    ("u2", "South"),
    (None, "Unassigned"),
], "user_id string, region string")

enriched = latest.join(users, "user_id", "left")
summary = (
    enriched.groupBy("region")
    .agg(
        F.count("*").alias("orders"),
        F.count("amount").alias("known_amounts"),
        F.sum("amount").alias("total"),
    )
    .orderBy(F.col("region").asc_nulls_last())
)
summary.show()

Expected output:

region | orders | known_amounts | total
North  | 2      | 2             | 205
South  | 1      | 0             | NULL
NULL   | 1      | 1             | 50

The null key on o4 does not match the null key in users under ordinary equality. The output region stays null rather than becoming "Unassigned". Null-safe equality is available when matching nulls is genuinely required; it would change this task's meaning. Spark null semantics

South has an order but no known amount. Its count is one, known-amount count zero, and sum null. Converting that sum to zero would assert information the input does not supply. The separate counts let a reader distinguish missing measurement from measured zero.

The dimension is unique on non-null user identifiers in this fixture. Verify that precondition before treating a left join as one output row per order. If the reference data has multiple matches, resolve its grain before enrichment rather than deduplicating the final result blindly.

Assert values, ordering, and schema separately

An expected output is useful only if the test checks the distinctions you care about. Build the expected DataFrame with explicit types and compare sorted results when output order is part of the contract.

expected = spark.createDataFrame([
    ("u1", "o1", 125),
    ("u1", "o2", 205),
    ("u2", "o3", None),
], "user_id string, order_id string, running_amount long")

assertDataFrameEqual(
    running, expected,
    checkRowOrder=True, rtol=0, atol=0,
)
assert running.schema == expected.schema

The versioned testing API defaults to ignoring row order and nullable differences. Here, row ordering is checked explicitly, numerical tolerances are zero, and direct schema equality separately checks the full schema. Choose those settings intentionally rather than assuming every property is checked by default. assertDataFrameEqual documentation

For other outputs, decide whether nullable metadata is part of your interface contract. Aggregations can produce metadata that differs from a manually declared expected schema even when values and column types agree. Document that distinction; do not weaken a test merely to turn a failure green.

The local verification also checked all four result sets, the four-row enriched count, the five-row outer expansion, and the three-row ordinary expansion. A negative example confirmed the alternative 205/205 range result. Those checks catch different mistakes from a single total-row-count assertion.

Explain edge cases before discussing performance

Try changing one input at a time. Add a later revision to o2, remove all tags from o1, or replace o3's unknown amount with zero. Predict exactly which expected rows change before rerunning. A zero should change South's known-amount count and total; it should not change its order count.

Next, vary physical input order and partitioning while preserving the logical records. The survivor and ordered final values should remain consistent because the preference rules are explicit. Such checks examine deterministic semantics, not whether one local run proves distributed reliability.

Keep collection and full expected DataFrames confined to small fixtures. A production-sized dataset should not be copied into driver memory just to imitate this test. Separate small transformation tests from larger integration checks with appropriate aggregate invariants and resource limits.

Once correctness is established, use our Spark shuffles, partitions, and debugging guide for execution-plan reasoning. These exercises do not establish an ideal partition count, broadcast threshold, or streaming deduplication policy.

These verified PracHub records train related reasoning. Some use SQL or general Python rather than PySpark; translate the relevant subproblem into DataFrame operations without claiming the original prompt requires Spark.

PracHub question	Practice focus
Reason About Duplicate Data and Scaling in Spark	Define identity and deterministic survivor rules before discussing scale.
Write SQL to rank top products per category	Translate group ranking and stable tie-breaking into a window.
Flatten nested JSON into a string map	State the nested input and output shape before choosing a flattening approach.
Aggregate exam scores with NULL handling	Explain missing-value rules before joining and aggregating.
Write SQL window functions for D7 retention	Define time boundaries, user grain, and ranking semantics explicitly.

Choose a Data Engineer practice prompt, write a tiny adversarial fixture, and predict the output on paper. Then implement the transformation and explain every mismatch. That rehearsal makes your answer inspectable instead of dependent on remembering a familiar code pattern.

Sources and Further Reading

Documentation checked September 8, 2026. Inputs and expected outputs are original practice material; local execution confirms the stated version's fixture results, not employer interview requirements.