Learning from Wrong Data

## How to Approach - Use STAR structure: be specific, quantify, own the mistake, and highlight learnings and systemic improvements. - Show strong product judgment: validate data, triangulate sources, apply guardrails, favor experiments for causal claims. - Keep it concise but concrete: 60–90 seconds of setup; the rest on detection, impact, and safeguards. ## Example Answer (PM Context) Situation - I owned monetization for a consumer product. Our weekly dashboard showed a 15% drop in ROAS (Return on Ad Spend) for a large cohort. ROAS = Revenue / Cost. - The dashboard indicated ROAS fell from 2.0 to 1.7 (−15%). Task - Protect margin quickly. I decided to pause a set of long‑tail campaigns that looked least efficient. Action - I paused ~20% of those campaigns and reallocated budget toward better performers. - Two days later, I noticed total daily revenue fell about 3% without the expected improvement in margin. - I initiated a reconciliation: compared our warehouse metrics with source platform totals and finance billing data. Discovery - We found the ETL had been updated to join a new cost adjustment table. Some ad groups had multiple cost rows per day (late-adjusted costs). The join duplicated cost records, inflating cost and depressing ROAS. - Quick check: In the affected cohort, platform-reported cost was $500k, but our warehouse showed ~$1M. With revenue stable at ~$1M, true ROAS ≈ 1.0M/0.5M = 2.0, not the 1.7 we saw. Impact - We paused viable campaigns for ~5 days: - Short-term: ~3% revenue dip for the week; some ramp-up friction to restore spend. - Team time: engineering + analytics spent 2 days on root cause and backfill. - I owned the decision, published a postmortem, and restored campaigns with a controlled ramp. Safeguards Implemented 1. Data quality and reconciliation - Primary-key and duplication checks on joins (assert one-to-one or one-to-many with bounded multiplicity). - Daily reconciliation to platform control totals with a 1% tolerance; alert if exceeded. - Freshness and null-rate monitors on critical fields (campaign_id, date, cost, revenue). 2. Decision guardrails - For material spend changes: require either (a) triangulation from two independent sources or (b) an experiment/holdout. - Introduced a stop-loss: automatic rollback if revenue or margin deviates >2% intraday without causal explanation. - Canary changes with a 10% traffic slice before global rollout. 3. Experimentation hygiene - When feasible, use randomized holdouts and guardrail metrics (e.g., conversion rate, refund rate) with sequential monitoring. - Stratified analysis by channel/geo to avoid Simpson’s paradox. 4. Process and accountability - Data contract for cost tables (unique keys, allowed late-arrival behavior) and code review checklist for metric changes. - Decision log capturing data sources used, assumptions, and pre-agreed reversal criteria. Result - Corrected ROAS returned to ~2.0; restored campaigns recovered revenue. - The new checks later caught a separate undercount within hours, preventing another bad decision. ## Why This Works - It shows ownership, fast detection, measured rollback, and systemic fixes. - It demonstrates PM judgment: don’t act on a single dashboard; reconcile and experiment for causal decisions. ## Small Numeric Illustration (Teaching Aid) - Suppose revenue = $1,000,000 and true cost = $500,000 → ROAS = 2.0. - A join duplicates cost rows to $1,000,000 → ROAS appears to be 1.0 (−50%). - Even a smaller duplication (true cost $500k, reported $588k) yields ROAS 1.7 (≈−15%), enough to trigger an overreaction. ## Pitfalls to Call Out - Causation vs correlation: don’t reallocate spend based on correlational drops without a holdout. - Aggregates hide heterogeneity: do stratified cuts (channel, geo, device) to avoid Simpson’s paradox. - Data freshness and backfills: late-arriving data can temporarily mislead; label dashboards with freshness SLAs. ## Template You Can Reuse - Situation/Task: We saw metric X drop/improve by Y% and made decision Z to protect/capture value. - Action: Specific steps you took and why the data seemed credible at the time. - Discovery: The exact flaw (instrumentation bug, duplication, attribution window, timezone, sampling bias) and how you detected it (reconciliation, A/A test, segment analysis). - Impact: Quantify user, revenue, or time cost. - Safeguards: Data quality checks, decision guardrails, experimentation standards, process changes, and a concrete example of those safeguards catching an issue later.