## Reason About Duplicate Data and Scaling in Spark In a technical experience discussion, explain how you would identify and handle duplicate data in a Spark pipeline and how you would scale the pipeline as data volume and traffic grow. No schema, latency target, or duplicate definition is supplied, so make those assumptions explicit before choosing operations. ### Part 1 — Define Duplicate Semantics Distinguish exact duplicate rows from repeated events, retried deliveries, and different records that share a business key. Define which copy should survive and for how long deduplication state must be retained. #### What This Part Should Cover - A stable event ID or a documented composite business key. - Full-row, key-only, and key-plus-time duplicate definitions. - A deterministic survivor rule for conflicting copies. - Batch versus streaming scope, late arrival, and replay semantics. ```hint Define identity before removing rows Two records that compare equal in one column are not necessarily the same event, while byte-identical retries may still need one logical outcome. ``` ### Part 2 — Design the Spark Processing Plan Describe the transformations, partitioning, and state needed for the chosen batch or streaming contract. Address shuffle cost, skew, and deterministic results. #### What This Part Should Cover - Schema validation and normalization before computing keys. - An exact operation plan, such as key-based aggregation, window ranking, or bounded streaming deduplication. - Partition keys and shuffle behavior for large or highly skewed keys. - Checkpoint, watermark, or state-retention behavior where streaming is used. ```hint Choose the survivor explicitly A deduplication job is incomplete until ties are resolved by a stable ordering rather than by whichever partition finishes first. ``` ### Part 3 — Scale and Operate the Pipeline Explain how the design evolves from a small batch job to sustained large-scale processing. Include retries, idempotent output, backfills, resource tuning, and observability. #### What This Part Should Cover - Incremental reads and partition pruning instead of rescanning all history. - File sizing, parallelism, memory pressure, shuffle spill, and skew mitigation. - Idempotent or transactional sink behavior under task and job retries. - A backfill plan that does not corrupt live output. - Metrics for input/output counts, duplicate rate, lag, state size, skew, spill, and failures. ```hint Scale the data movement first More executors do not fix a plan that rereads all history, sends one hot key to one task, or rewrites the same output on every retry. ``` ### What a Strong Answer Covers - A business-level duplicate definition translated into deterministic Spark operations. - Correct batch or streaming state boundaries and late-data behavior. - Explicit shuffle, partitioning, skew, and retry semantics. - Reproducible backfills, idempotent output, and measurements that reveal both correctness and performance drift. ### Follow-up Questions 1. How would you choose a watermark when duplicates can arrive days late? 2. What should happen when two records share an event ID but disagree on payload? 3. How would you diagnose one key that dominates a shuffle partition? 4. How would you prove that rerunning a failed partition does not duplicate sink output?