Demonstrate business impact from a project

# How to Answer, Plus a Worked Example This format earns high marks in a technical screen: - 1–2 sentences for business context and decision. - 2–3 sentences to define the OEC (overall evaluation criterion) and guardrails. - Measurement plan and assumptions; how you validated them. - Numbers: baseline, lift, sample sizes, confidence interval, and dollars. - Stakeholders, risks/guardrails, adoption. - Timeline, trade‑offs, what failed, and the retrospective. Below is a worked example tailored to a two‑sided marketplace. Numbers are illustrative but realistic and show the level of depth expected. ## Example Project: Reducing Rider Cancellations via Improved Dispatch Scoring 1) Problem and Decision - Problem: High rider cancellation rate during peak demand and congested corridors led to lost trips and poor experience. Internal analysis showed cancellations correlated with long and volatile pickup ETAs and suboptimal driver assignment. - Decision enabled: Whether to launch a new dispatch scoring function that penalizes high‑variance pickup ETAs and slightly expands the candidate driver set in congested areas. 2) Metrics (Definitions) - Primary OEC: Trip Completion Rate (TCR) = completed trips / requests. - Guardrails: - Pickup ETA (mean/95th percentile). - Driver cancel rate and deadhead distance. - Driver earnings per online hour. - Surge minutes share (marketplace stability). 3) Counterfactual and Dollar Impact - Measurement window: 14‑day geofenced online experiment across 3 large cities. - Sample sizes: n_treat = 6,000,000 requests; n_control = 6,000,000 requests. - Baseline (control): cancellation rate p_c = 8.00% → TCR_c = 92.00%. - Treatment: cancellation rate p_t = 7.62% → TCR_t = 92.38%. - Lift: ΔTCR = +0.38 percentage points (pp) absolute. Confidence/uncertainty (difference in proportions): - Δp = p_c − p_t = 0.0800 − 0.0762 = 0.0038. - Standard error: SE(Δp) = sqrt[p_t(1 − p_t)/n_t + p_c(1 − p_c)/n_c] = sqrt[(0.0762×0.9238)/6e6 + (0.08×0.92)/6e6] ≈ 0.000155. - 95% CI: 0.0038 ± 1.96×0.000155 ≈ [0.00350, 0.00410] → [0.35 pp, 0.41 pp]. Dollar impact (illustrative unit economics): - Assume contribution margin per completed trip (after variable costs) CM ≈ $1.10. - Monthly requests in the 3 test cities ≈ 36,000,000. - Incremental completed trips/month ≈ 36,000,000 × 0.0038 = 136,800. - Dollar impact/month ≈ 136,800 × $1.10 = $150,480. - 95% CI using Δp bounds: $138,600 to $162,300 per month (test cities). Scaling (with prudence): - Suppose the broader network sees ~400,000,000 requests/month and we apply a 0.7 shrinkage factor for heterogeneity/operational differences. - Scaled impact ≈ 400,000,000 × 0.0038 × $1.10 × 0.7 ≈ $1.16M/month. - With CI on Δp, a plausible range is roughly $1.07M–$1.25M/month, acknowledging additional uncertainty from the shrinkage factor. 4) Measurement Strategy - Design: Cluster‑randomized online A/B test to limit interference (drivers and riders interact). We randomized by hex‑grid geofences so a rider is always served under a consistent policy within a cell. Drivers were pinned to the policy of the pickup cell at assignment time. - Why experiment: Direct counterfactual with high traffic; avoids bias from time trends and confounding supply shifts. - Variance reduction: CUPED used rider‑level and cell‑level pre‑period cancellation rates as covariates; stratified randomization by hour‑of‑day and cell density. - Assumptions and validations: - Interference minimized: Chose sufficiently large cells; border analysis showed no significant spillovers; sensitivity excluding border cells unchanged. - Randomization balance: A/A checks and covariate balance (pickup ETA distribution, request mix, weather/events) passed. - Stable logging: A/A period showed no metric drift; counter logs for dispatch decisions matched server truth ≥ 99.9%. - Seasonality captured: 14 days included 2 full weekends; pre/post by day‑of‑week consistent. Quasi‑experimental fallback (if A/B impossible): - Matched diff‑in‑diff using synthetic control at the cell level with pre‑period trends; instrumented by policy availability windows. Validated via placebo tests and parallel‑trends checks. 5) Execution, Stakeholders, and Adoption - Stakeholders: Product (throughput), Operations (driver experience), Engineering (reliability), Finance (unit economics), Policy/Support (edge cases). - Misalignment/pushback: - Ops worried about longer deadhead and driver satisfaction; Finance needed margin proof, not just TCR. - Response: Agreed on an OEC with guardrails and a hard cap: mean pickup ETA could not worsen by > 0.1 min; driver deadhead +< 0.5%; earnings/hour within ±0.25%. - Risks and guardrails: - Real‑time monitors for pickup ETA, driver cancel rate, deadhead distance, and incident flags; auto kill‑switch if any guardrail breached for 15 consecutive minutes per city. - Phased rollout: 10% → 25% → 50% → 100% with holdouts for continued monitoring. - Adoption: Shared weekly readouts, city‑level playbooks, and a rollback plan. PM and Ops co‑owned rollout gates; Finance validated the CM assumptions and signed off on the dollar impact. 6) Timeline, Trade‑offs, and Retro - Timeline (approx.): - Weeks 1–2: Root‑cause analysis and metric design; offline simulation from historical data. - Weeks 3–5: Feature engineering and model/scoring changes; load testing and logging hardening. - Weeks 6–7: A/A and small city pilot to validate instrumentation and guardrails. - Weeks 8–9: 14‑day cluster A/B across 3 cities. - Week 10: Analysis, decision review, and rollout plan. - Trade‑offs: - Accepted a small deadhead increase (+0.4%) to gain +0.38 pp TCR; tightened the variance penalty to keep pickup ETA Δ within +0.06 min. - Slightly lower surge minutes (−0.2 pp) from smoother matching, a positive side‑effect for user experience but requiring pricing team alignment. - What failed/surprised us: - Offline replay overstated gains (~0.8 pp predicted vs. 0.38 pp realized) due to unmodeled driver rejection behavior and road incidents; we fixed by incorporating uncertainty in pickup ETA and adding a driver‑acceptance model in simulations. - Early logging missed a rare reassignment path; A/A caught it and we patched before the main test. - What I’d do differently: - Start with cluster A/A and border‑spillover diagnostics earlier to quantify design effect and power needs. - Pre‑register the analysis and CUPED covariates to reduce researcher degrees of freedom. - Build an automated generalization/shrinkage pipeline for rollout forecasts (learned effect modifiers: density, congestion, driver supply volatility). ## Teaching Notes and Templates You Can Reuse - Counterfactual/dollars template: - Impact$ = Requests × Δ(primary metric) × Contribution margin per unit. - Use CI bounds on Δ to report a range; apply a shrinkage factor for generalization beyond test scope. - CI for difference in proportions: - 95% CI: Δ ± 1.96 × sqrt[p_t(1−p_t)/n_t + p_c(1−p_c)/n_c]. - Power planning (back‑of‑the‑envelope): - For absolute effect d with baseline p and equal arms, n per arm ≈ 2 × (z_{1−α/2} + z_{1−β})^2 × p(1−p) / d^2. Adjust by design effect for clustering: DE = 1 + (m−1)×ICC. - Guardrail playbook: - Define thresholds ex‑ante and wire kill‑switches; monitor in real time; keep a holdout group until post‑launch stability is proven. This level of specificity—clear decision, precise metrics, validated counterfactual, quantified dollars with uncertainty, and evidence of leadership—meets the bar for a Data Scientist technical screen.