Design metrics and experiment for stolen-post detection

You work on **Stolen Post Detection** for a social platform (detecting content that is copied/reposted without permission). A new detection algorithm is proposed (e.g., a model producing a stolen-probability score used to downrank, label, or block posts). ## Questions 1) **Problem framing & diagnostics** - What are the key failure modes and risks (false positives vs false negatives) for stolen-post detection? - If stakeholders report “stolen posts are down,” what would you check to validate whether this is real vs an artifact (measurement issues, reporting changes, seasonality, policy changes, spam shifts, etc.)? 2) **Metrics** Propose: - **Primary success metric(s)** (what you ultimately want to improve) - **Diagnostic metrics** (to understand why things moved) - **Guardrail metrics** (to prevent harm) Include at least one metric that handles delayed / noisy ground truth (since “stolen” labels may come from user reports, manual review, or appeals). 3) **Experiment design** Design an online experiment (A/B test or alternative) to evaluate the new algorithm. Address: - Randomization unit (post-level vs author-level vs viewer-level) and why - Interference / network effects (e.g., copied content affects multiple creators) - Exposure definition (who is affected by the change) - Sample size / power considerations at a high level (what drives variance) - Ramp plan and decision criteria 4) **Tradeoffs and decision** If offline metrics improve (e.g., higher precision/recall on labeled data) but online engagement drops, how would you decide what to launch and what follow-ups you’d run?