Evaluating the Impact of Duplicate and Stolen Posts on a Content Platform

## Evaluating the Impact of Duplicate and Stolen Posts on a Content Platform You are a data scientist at a large user-generated-content platform (think a social feed where users publish posts and others engage with them). Product leadership is worried about **duplicate posts** (the same content posted more than once, sometimes by the same author, sometimes re-uploaded by others) and **stolen posts** (one user re-publishing another user's original content as their own, without credit). The platform is considering shipping detection-and-enforcement systems (deduplication, "stolen content" takedowns, attribution back to the original author). Your job is to design how the company would **measure the impact** of duplicate and stolen posts — and of any intervention against them — on the health of the platform. This is a multi-part case. Work through each part in order. ### Constraints & Assumptions - The platform has tens of millions of daily active users and millions of posts created per day; content is a mix of text and media. - You can instrument arbitrary client and server events and run randomized experiments. - A content-similarity / near-duplicate detection service exists or can be built (hashing + embeddings); it returns a similarity score, not ground truth. - "Impact" is ultimately about long-term ecosystem health (original-creator supply and consumer retention), not a single short-term click metric. ### Clarifying Questions to Ask - What is the platform's primary business goal here — protecting original creators (supply side), improving consumer experience (demand side), or legal/compliance risk from stolen IP? The metric hierarchy changes depending on which. - Are duplicate posts and stolen posts being treated as the same problem or two different problems? (Self-duplication vs cross-author theft have different harms and different fixes.) - What enforcement action is on the table — demotion in ranking, hard removal, watermark/attribution to the original author, or creator-facing warnings? The intervention determines what we can experiment on. - What is our tolerance for false positives? Wrongly removing an original author's content is far costlier than missing one thief. - Over what horizon does leadership want the impact measured — a 2-week experiment readout, or long-term creator retention over months? - Do we have reliable ground-truth labels (e.g., human-reviewed cases or DMCA reports) to validate any automated "stolen" classifier against? ### Part 1 — Choosing metrics Propose the set of metrics you would use to quantify the impact of duplicate and stolen posts on the platform. Organize them so leadership understands what each one is for, and identify which single metric (or small set) you would treat as the primary success metric versus guardrails. ```hint Where to start Build a small metric tree. Separate prevalence/quality metrics (how much duplicate or stolen content exists), creator-side metrics (do original authors keep producing?), and consumer-side engagement/retention metrics. Then map each to the mechanism by which stolen content hurts the platform. ``` ```hint Primary vs guardrail The thing leadership ultimately cares about is long-run ecosystem health: original-creator retention and consumer engagement/retention. Prevalence ("% of posts flagged duplicate") is a diagnostic, not a north star — it can be gamed by changing the detector's threshold. Think about whether a *share-of-impressions* version is more decision-relevant than a *count* version. ``` #### What a Strong Answer Covers - A structured **metric tree** that separates prevalence/quality, creator/supply-side, and consumer/demand-side metrics, with each metric tied to a *mechanism* of harm. - An explicit choice of **primary metric(s) vs guardrails**, with the false-positive rate on originals treated as a hard guardrail. - Recognition that **prevalence is a gameable diagnostic** (threshold-dependent), not a north star, and that share-of-impressions/engagement is often more decision-relevant than raw counts. ### Part 2 — Defining "stolen posts" Write down an operational definition of a **stolen post** that an engineering system could actually compute at scale, then critique it: what are the failure modes of your own definition (false positives and false negatives)? ```hint Construct the definition A workable definition needs (a) a content-similarity test (near-duplicate detection via hashing/embeddings over text or media) and (b) an originality/ownership test (who posted the substantially-similar content first, accounting for cross-account re-uploads). ``` ```hint Where the definition breaks Stress it against legitimate behavior: quotes, reaction/duet/stitch formats, memes and templates, news everyone reports on, licensed reposts, the original author reposting their own work, and back-dated content imported from another platform where "first seen here" is not "created first." ``` #### What a Strong Answer Covers - An **operational, computable** definition combining a similarity test and an originality/ownership test (substantially-similar, posted later, different author, no attribution). - A concrete inventory of **false-positive sources** (quotes, reactions/duets, memes/templates, commodity news, licensed/self reposts) and **false-negative sources** (paraphrase/translation, cropped/re-encoded media, sub-threshold similarity, cross-platform imports). - Awareness that the definition is **threshold-dependent and cost-asymmetric** (harming an original ≫ missing a thief), motivating softer actions or human review in ambiguous cases. ### Part 3 — Designing the experiment You want to ship a "stolen-content suppression" intervention (demote or remove stolen reposts and attribute originals) and measure its causal effect. Design the experiment. The interviewer explicitly flagged that **network effects** complicate this — explain what the network effect is here and how it threatens a naive user-level A/B test, then propose a design that addresses it. ```hint Name the threat A standard user-level A/B test rests on an assumption about isolation between units. What is that assumption, and why does it break when you suppress a piece of shared content? ``` ```hint Direction of the bias If control users are partly affected by the treatment (because they consume or produce content that is also consumed by treated users), think about which direction that contamination pushes the measured treatment-minus-control gap. ``` ```hint Designs that contain interference The core idea is to find a randomization unit large enough that most interactions stay within a single arm. What unit choices exist on a content platform, and what does each give up in terms of power or residual leakage? ``` #### Clarifying Questions for this Part - How self-contained are our communities/geos — do most views and reposts stay within a region or language, or does content cross freely? This determines whether cluster randomization actually bounds spillover. - What effect size does leadership consider meaningful, and how long can the experiment run? Clustering inflates variance, so the minimum detectable effect (MDE) and runtime budget drive the design choice. #### What a Strong Answer Covers - A correct, named diagnosis: **network effects = interference / SUTVA violation**, with a clear mechanism for how treatment leaks into control through shared content. - A reasoned statement of the **direction of bias** (contamination shrinks the estimate toward zero) and the practical consequence. - At least one **interference-robust design** (cluster/community/geo, ego-network, creator/content-level, or switchback) with explicit **bias-vs-variance / power trade-offs**, plus pre-registration of the primary metric, an MDE/power check, and a variance-reduction plan. ### Part 4 — Interpreting a metric drop The experiment ships to the treatment arm and you observe that **one of your engagement metrics dropped** in treatment relative to control. Walk through how you would diagnose *why* — enumerate the plausible explanations (both "the intervention is genuinely bad" and "the metric is misleading") and how you'd distinguish them. ```hint Two buckets Split causes into (1) the metric fell for a *good* reason — you removed low-quality/stolen content that was generating hollow engagement, so a raw volume metric drops while value per session rises; and (2) the metric fell for a *bad* reason — over-suppression, false positives hitting originals, latency/UX regression, or a measurement/instrumentation bug. ``` ```hint How to adjudicate Segment the drop (is it concentrated on flagged content vs all content? new vs returning users? specific surfaces?), check whether guardrail/quality metrics move the opposite way, look at the false-positive rate of the detector, and watch the time trend for novelty effects. ``` #### What a Strong Answer Covers - A disciplined refusal to read a single metric literally, sorting hypotheses into a **"good drop"** bucket (removed hollow engagement; engagement reallocated to originals) and a **"bad drop"** bucket (over-suppression/false positives, feed-quality hole, UX/latency or instrumentation bug). - A concrete **diagnostic playbook**: segment the drop, read guardrails and quality-weighted metrics, inspect the detector's false-positive rate, and check the time trend for novelty effects. - A **decision rule** that ships only when primary metrics and guardrails are healthy, treating a raw-volume drop alone as non-blocking — and awareness of pitfalls like Simpson's paradox in aggregated comparisons. ### What a Strong Answer Covers These dimensions span all four parts and are what separate a strong candidate across the whole case: - **Mechanism-first thinking** — every metric, definition, and design choice is justified by *how* duplicate/stolen content actually harms the platform (supply, demand, trust), not by default best-practice. - **The cost asymmetry** between harming an original creator and missing a thief, carried consistently from metric choice (FP guardrail) through definition (precision bar) to interpretation (false-positive checks). - **Causal-inference rigor** — confounders, selection bias, SUTVA/interference, novelty/primacy effects, and the difference between correlation and a clean causal readout. - **Intellectual honesty** — naming the limits of one's own definitions and metrics (threshold-dependence, gameability) rather than presenting them as ground truth. ### Follow-up Questions - Suppose creator-supply metrics improve but short-term consumer engagement drops, and the two never reconcile within the experiment window. How do you make a ship / no-ship recommendation under that tension? - Your "stolen post" classifier has 95% precision. Is that good enough to auto-remove content? Quantify the expected number of wrongly-removed original posts per day at your platform's scale and discuss the policy implication. - The team proposes only **demoting** (not removing) stolen reposts. How does that change your experiment design, your metrics, and your interpretation of an engagement drop? - How would you detect and adjust for a **novelty effect**, where treated users react to the visible change itself rather than to the long-run steady state?