Navigate urgency, priorities, and conflict

# Context Assume a Data Scientist role in a large two‑sided mobility marketplace (rider–driver). The project spans marketplace health, pricing, and reliability, with cross‑functional partners in Product, Engineering, Operations, Finance, Legal/Policy, and Customer Support. ## (a) Why this team and why now - Mission tie: I’m motivated by building reliable, equitable transportation at scale. The marketplace’s goal—minimizing wait time while sustaining driver earnings—maps to my core skills in causal inference, experimentation, and production analytics. - Metrics I’d own: median/95th percentile wait time (ETA), trip completion rate, cancellation rate, dispatch success rate, and driver earnings fairness (variance and Gini). - Domain challenges that excite me: - Spiky, non‑stationary demand with geographic heterogeneity. - Real‑time decisioning under latency constraints (sub‑100 ms scoring, data freshness SLAs). - Two‑sided incentives with fairness and regulatory constraints. - Observability and incident response for mission‑critical marketplace services. ## (b) Project: Dynamic Incentives to Stabilize Peak Demand - Goal: Reduce peak‑hour wait time by 10% and cancellations by 3% in two metros by launching real‑time, geo‑targeted driver incentives informed by marketplace health signals. - How I got staffed: I had previously shipped a causal attribution framework for driver supply changes. Product tapped me to lead analytics and experimentation for this initiative. - Stakeholders: - Internal: Product (Marketplace, Pricing), Engineering (Incentives Service, Data Infra), Operations (city teams), Finance (budget), Legal/Policy (fairness/compliance), Customer Support (rider/driver sentiment), Data Platform. - External: Driver councils (feedback), select B2B partners (scheduled rides). - My role: - Define success metrics and guardrails; design the experimental rollout; estimate budget/ROI; build monitoring dashboards; partner with Eng to instrument events; lead the analysis/readout. - What was hard and why: - Ambiguous causal pathways: Incentives affect driver supply, which shifts surge and ETA; confounded by weather/events. - Data latency and reliability: Health signals needed sub‑minute freshness; some sources updated every 5–15 minutes. - Governance: Ensuring geographic fairness and avoiding unintended earnings volatility. ## (c) Sev‑1 incident and 48‑hour deliverable Midway through the pilot, a regression in the surge model under‑priced certain zones during a large event. Symptoms: +15% median wait time, +8% rider cancellations in two metros; support tickets spiked. Leadership requested, within 48 hours: (1) quantified impact, (2) mitigation plan, (3) provisional recalibration. - My prioritization framework across four concurrent projects: 1) Ongoing dynamic incentives A/B test (high impact/time‑sensitive). 2) Rider churn model refresh (high impact/less urgent). 3) Fraud anomaly PoC (medium impact/medium urgency). 4) Weekly city demand forecast (operational, routine). I scored work using: Severity × Blast Radius × Reversibility, plus time‑to‑mitigation and alignment to company KPIs. - Tradeoffs (quality vs. time): - For incident impact sizing, I used a fast Difference‑in‑Differences (DiD) with a nearby control city unaffected by the regression, rather than a full causal forest with heterogeneity. - DiD estimator: (Y_treat_post − Y_treat_pre) − (Y_ctrl_post − Y_ctrl_pre) - Example: If wait time (min) = 5.6→6.5 (treat) and 5.4→5.5 (control), DiD = (6.5−5.6) − (5.5−5.4) = 0.9 − 0.1 = +0.8 min impact. - Cut: advanced heterogeneity by zone and driver‑cohort; long‑run churn estimates; full backfill with late‑arriving data. - Defer: churn model refresh by one sprint; fraud PoC experiments by 1 week. - Parallelize: One analyst pulled event‑level logs; I ran DiD and cost impact; Eng owned rollback/feature flag; Ops compiled qualitative support signals; Finance validated variable spend. - Communication and alignment: - Set up a 24‑hour “war room” with a 1‑pager: incident summary, hypotheses, decision log, mitigation plan, and a risk register (likelihood × impact, owner, next check‑in). - Executive updates at T+12h and T+36h: current impact estimate, confidence, mitigation status, and next steps. - Guardrails for mitigation: ensure rider wait time <= target +5%, driver earnings variance change < 2 p.p., and no geography’s incentives exceed budget cap. - Outcome at T+48h: - Rolled back the faulty model; applied a temporary surcharge floor in the affected zones. - Estimated incremental impact: +0.8 min to median wait, −2.9 p.p. completion; projected revenue loss $420k (95% CI: $350k–$500k). Confidence: medium (control city match validated via pre‑trend checks, p>0.1 for pre‑trend difference). ## (d) Disagreement on methodology and scope - Disagreement: Product wanted to declare success based on pre‑post improvements in the pilot city and scale globally. I insisted on city‑level randomized rollouts or, at minimum, DiD with pre‑trend validation and power analysis. - How I surfaced assumptions: - Wrote assumptions explicitly: seasonality, event confounders, driver supply spillovers, regression to the mean. - Showed a counterexample where a pre‑post view “improved” due to a weather shift absent any treatment. - Evidence gathered: - Back‑tested the incentive policy using historical weeks (off‑policy evaluation via inverse propensity weighting using prior propensity to be incented by zone/time). - Ran a 2‑city stepped‑wedge rollout with 20% zone‑level randomization. - Power analysis for the primary metric (median wait time). For a target detectable effect δ = 0.4 min, baseline σ ≈ 2.0, cluster‑robust ICC ≈ 0.05, we computed the required number of zone‑hours to reach 80% power at α = 0.05; we met it in 10 days. - Influence without authority: - Framed the trade: “Faster launch” vs. “Credible, scalable proof” with quantified risk of false positives (type I error ~30% under plausible confounding). - Proposed a compromise: limited ramp with holdout zones and a 7‑day readout gate. - Resolution and metrics moved: - Agreement to run the stepped‑wedge with guardrails. - Results: −7.2% median wait time (−0.42 min), −3.1 p.p. cancellations, +4.3% driver online hours; budget +1.6% variable spend. Key guardrails within limits. p<0.01 for primary outcomes; no significant negative movement in earnings variance. ## (e) Outcomes, lessons, and institutionalization - Outcomes: - Shipped dynamic incentives to two metros; expanded to four after 6 weeks. - Built a near‑real‑time marketplace dashboard: wait time percentiles, dispatch rate, cancellation rate, surge accuracy, incentive take‑rate; with alerting on z‑score anomalies and data freshness SLIs. - Documented the incident, including root cause (model regressor drift + insufficient shadow testing) and time‑to‑detect (TtD) reduction plan. - What I’d do differently: - Introduce shadow deployments for pricing/surge models with automatic canary analysis before full enablement. - Pre‑register experiment designs and analysis plans to align stakeholders earlier. - Tighten data contracts and freshness SLAs for health signals feeding incentives. - How I’d institutionalize improvements: - Playbooks: Incident response runbook (Sev‑1/2), with roles, first‑hour checks, standard queries, DiD templates, and a communication cadence. - Dashboards & alerts: SLOs for surge accuracy and ETA calibration; freshness monitors for critical Kafka topics with page‑on‑breach; guardrail alerting on cancellations and earnings variance. - SLAs & ownership: Clear RACI for model changes (DS sign‑off, Eng owner, Product approver); pre‑launch checklist (instrumentation parity, shadow metrics within tolerance, rollback plan tested). - Experiment standards: City/zone‑level randomization defaults; guardrail metrics hard‑coded in experiment config; minimum detectable effect calculators embedded in experiment tooling. - Validation/guardrails summary: - Primary: median wait time, completion rate. - Guardrails: cancellation rate, driver earnings variance, surge error (|actual − predicted|), support ticket rate, data freshness SLI. - Post‑launch: Weekly DiD readouts; heterogeneity checks (zone, hour, cohort); counterfactual simulations under demand spikes. This end‑to‑end approach demonstrates handling ambiguity, coordinating across teams, making principled speed‑vs‑quality tradeoffs under pressure, and converting lessons into durable processes and tooling.