Demonstrate Leadership in Ambiguous Analytics Projects

# Example Answer and Teaching-Oriented Walkthrough Below is a structured, end-to-end example suitable for a marketplace analytics role. It demonstrates scoping under ambiguity, stakeholder management, methodological rigor, and execution speed. ## 1) Situation/Task - Context: A major city was heading into a holiday weekend expected to spike demand. Recent trends showed rising rider cancellations attributed to long ETAs and surge price spikes. Leadership needed a go/no-go decision in 10 days on a set of levers (targeted rider promos, temporary driver incentives, and match-policy tweaks) to protect growth while avoiding supply burnout. - Ambiguity: Different teams pushed different definitions of success: Growth favored request volume; Marketplace favored supply health and reliability; Support emphasized complaints. Goals weren’t reconciled. - Success criteria (finalized in a working session): - Primary KPI: Incremental completed trips (ICT) vs. counterfactual. - Guardrails: - Cancellation rate: −10% or better. - p90 ETA: improve by ≥ 1 minute (avoid long-tail wait times). - Driver earnings per online hour (DEPH): ≥ baseline (no supply harm). - Unit economics: contribution margin non-negative. - Decision deadline: 10 days to recommend a plan; 14 days to pilot and decide on broader rollout. ## 2) Action ### a) Scoping under ambiguity - Problem framing: I created a metric DAG linking upstream levers to outcomes: supply online hours → acceptance rate → ETA distribution → rider conversion → cancellations → completed trips → margin. This clarified failure modes and measurable touchpoints. - Hypotheses: 1) Long-tail ETAs (p90) drive outsize cancellations. 2) Broad demand promos without supply pacing worsen p90 ETAs, increasing cancellations despite higher requests. 3) Targeted promos in time/geo pockets with slack supply and light driver incentives yield net lift in completed trips without harming DEPH. - Data checks: Baseline by hour-of-week and zone; regression and partial dependence plots showing cancellation probability rising sharply above 9-minute ETA; heterogeneity by zone and weather. ### b) Prioritizing stakeholders and negotiating trade-offs - Stakeholders and objectives: - Growth: maximize short-term request volume. - Marketplace Ops: protect driver earnings and reliability (p90 ETA, cancellations). - Finance: maintain margin. - Support: reduce wait-time complaints. - Trade-off framework: - Proposed primary KPI (ICT) with guardrails (p90 ETA, DEPH, margin). Framed as: “We will grow only where reliability and supply health hold.” - Targeting: Limit promos to zones/hours with acceptance rate > 88% and idle time > 10% (proxy for slack supply). Pair with small driver incentives in thin zones. - Pacing: cap promo issuance per 15-minute interval to avoid demand spikes. ### c) Pushing back on a misleading metric (senior stakeholder) - Request: Senior Growth leader wanted to set success by average ETA (mean) and total requests, arguing they are simple and fast to move. - Why misleading: - ETA distributions were heavy-tailed; the mean masked reliability issues. A small reduction for many riders could coexist with a worse tail for some riders, increasing cancellations and complaints (Simpson’s paradox across zones). - Evidence presented: - Histogram of ETAs showed a long tail; pilot sim indicated mean ETA −8%, but p90 ETA +1.5 minutes in high-traffic zones, with a corresponding +3.2 pp cancellation rate in those zones. - Correlation: p90 ETA had 2.3× stronger correlation with cancellations than mean ETA; NPS drops aligned with tail deterioration. - Small numeric example: Two zones, equal volume. - Zone A: mean ETA 6→5.5 min (good), p90 10→9.5. - Zone B: mean ETA 8→7.2 (good), p90 12→14 (worse). Overall mean looks better, but cancellations rose due to Zone B’s tail. - Communication approach: - Pre-read with 1-page visual, highlighted customer harm stories tied to tail risk. - Framed as risk mitigation: “Optimizing mean risks hidden churn; p90 protects reliability.” - Proposed compromise: Primary KPI = ICT; reliability guardrail = p90 ETA; Growth gets visibility to requests as a leading indicator, not a success metric. - Outcome: Agreement to track p90 ETA and cancellations as guardrails; requests downgraded to a leading indicator. ### d) Experiment and analysis design - Design: Geo-time segmented rollout across matched city zones (synthetic control + difference-in-differences to estimate incremental impact while controlling for seasonality and weather). - CUPED adjustment: Reduced variance using pre-period outcomes X with Y* = Y − θ(X − μ_X), where θ = Cov(Y, X) / Var(X). - Power: Aimed to detect a +5% lift in completed trips at 90% power; variance estimates from 8 weeks of history guided zone sample selection. - Leading indicators monitored hourly: search-to-request conversion, acceptance rate, p90 ETA, driver idle time, DEPH, and wait-time complaint rate. ## 3) Result - Outcomes over 10-day pilot (test vs. control, CUPED-adjusted): - Incremental completed trips: +6.8% (95% CI: +4.9% to +8.6%). - Cancellation rate: −12.1% (from 16.5% to 14.5%). - p90 ETA: −1.3 minutes (from 11.2 to 9.9). - DEPH: +3.0% (maintained supply health). - Contribution margin: +1.1 pp (after promo and incentive costs). - Operational learning: On day 2, a zone breached p90 ETA guardrail due to a local event; auto-pacing throttled promos by 35% in that zone within 30 minutes, preventing broader tail deterioration. - Reproducibility and handoff: - Code: Versioned repo (SQL + Python), modularized queries, data contracts on key tables, unit tests for transformations. - Pipelines: Scheduled daily jobs (orchestration) to refresh metrics; anomaly checks on core KPIs with thresholds and backfills. - Dashboards: Executive view (ICT, cancellations, p90 ETA, DEPH, margin), Ops view (zone/hour cuts, alerts). - Docs: 2-pager experiment design and assumptions, metric definitions, lineage; runbook for incident response and tuning. - Alerts: Slack/email when guardrails breached (e.g., p90 ETA +1 min vs. baseline for ≥2 consecutive intervals), with auto-pacing hooks. ## 4) Reflection - What I’d change: I would pre-register heterogeneous treatment effects (HTE) by zone-type and ETA bands and use that to target from day 1, rather than learning mid-pilot. We discovered certain dense downtown zones benefitted less from promos without stronger supply incentives. - Early signal that would have prompted the change sooner: The first 24-hour uplift distribution showed high variance and a right-skew in cancellations for downtown zones. A Bayesian shrinkage view of zone-level lift would have flagged low posterior probability of positive lift earlier, justifying immediate re-targeting. ## 5) If re-running as a 2-week sprint (Day-by-Day) - Day 1: Kickoff, clarify decision deadline, align success metrics (ICT primary; p90 ETA, cancellations, DEPH, margin guardrails). Data audit and access checks. - Day 2: Metric DAG, baseline by zone/hour, define eligibility (zones with acceptance rate > 88%, idle time > 10%). Pre-register hypotheses and HTE plan. - Day 3: Experiment design (geo-time DiD + synthetic control), sample size/power, CUPED covariates, guardrail thresholds and auto-pacing rules. - Day 4: Build data models, QA tests, backfill 8-week baselines; create monitoring dashboard skeleton. - Day 5: Stakeholder readout to confirm trade-offs; finalize promo/incentive parameters; implement alerts. - Day 6: Soft launch in 2 pilot zones; validate data plumbing, latency, alert firing, and guardrail logic with dry runs. - Day 7–8: Expand to matched zones; hourly monitoring; adjust pacing in response to guardrail breaches. - Day 9–10: Interim analysis with CUPED; HTE cuts; decision checkpoint to keep/stop/retarget zones. - Day 11–12: Consolidate results, margin analysis, sensitivity checks (weather/events), falsification tests. - Day 13: Executive readout with recommendation; pre-approve scaled rollout conditions. - Day 14: Handoff: finalize dashboards, runbook, pipeline ownership; schedule post-rollout review. ### Interim leading indicators - Demand: search-to-request conversion, request growth (as signal, not success metric). - Supply: online hours, acceptance rate, driver idle time, driver cancellations. - Reliability: p50/p90 ETA, wait-time complaint rate. - Health: DEPH, contribution margin per trip. - Safety valves: Guardrail breaches trigger auto-pacing and/or zone-level pause. ## Methods Summary (for learning) - Incremental lift via DiD: Lift = (Y_treat,post − Y_treat,pre) − (Y_ctrl,post − Y_ctrl,pre). - CUPED variance reduction: Y* = Y − θ(X − μ_X), with θ = Cov(Y, X)/Var(X). - Tail-aware reliability: Optimize p90 ETA (and cancellations due to wait) instead of mean ETA to avoid masking tail risk. - HTE-first targeting: Use early zone-level posteriors to reallocate exposure quickly. ## Pitfalls and Guardrails - Pitfall: Vanity metrics (requests, mean ETA) can mask harm; use causal KPIs and reliability guardrails. - Pitfall: Spillovers between zones; use geo buffers and sensitivity checks. - Guardrails: Pre-commit to thresholds and automatic throttling to prevent runaway tail risk. - Validation: Placebo tests in pre-period, backtest on prior events, and monitor data quality alerts.