Solve a challenge using data

# Example Answer: Reducing Order Cancellations from Out-of-Stock Items ## 1) Decision Launch a real-time substitution and inventory-quality feature to reduce order cancellations due to out-of-stock (OOS) items in a same-day grocery marketplace. We had to decide whether to (a) launch to all stores, (b) launch only to high-OOS stores, or (c) not launch. Why it mattered: Cancellations drove refunds, shopper support contacts, and churn, directly impacting contribution margin and customer LTV. ## 2) Hypotheses - H1 (primary): Enabling real-time, model-driven substitutions and improved inventory signals reduces item-driven cancellations. - H2: It increases substitution acceptance rate without increasing shopper time per order. - H3: It improves contribution margin via fewer refunds and higher order completion. - H0: No meaningful changes beyond noise. ## 3) Success Metrics - Primary: Item-driven cancellation rate (cancellations per order due to OOS). - Secondary: Substitution acceptance rate, customer NPS/CSAT, per-order contribution margin. - Guardrails: Shopper time per order (+minutes), customer contact rate, refund rate, app crash rate. ## 4) Stakeholders - Product & Engineering: Feature design/implementation and experiment platform. - Operations (Shopper Ops, Retail Ops): Training and retailer coordination. - Finance: Margin, forecasting, financial controls. - CX/Support: Contact rates and resolution time. - Legal/Compliance: Messaging and consent for retailer feeds. ## 5) Data Sources - Orders and line-item events (add-to-cart, pick, substitute, refund, cancel). - Shopper app telemetry (time per item, substitution offers/accepts). - Retailer inventory feeds (where available) + historical sell-through. - Catalog and store metadata (availability flags, store traffic, hours). - Promotions/pricing tables and support tickets. - Experiment assignments and tracking. Data quality steps: deduplicated line items, reconciled event clocks, standardized reason codes, and created store-level OOS quality scores based on historical pick success. ## 6) Method: Geo-Experiment at the Store Level - Design: Cluster-randomized A/B test at the store level for 6 weeks. - Randomization: Within each retailer–state stratum, randomly assign stores to Treatment (new substitution + inventory-quality feature) or Control (status quo). This balances retailer mix and regional seasonality. - Sample Size: Target a 10% relative reduction in cancellations (baseline 6.0%, SD ≈ 2.5 pp across stores). Using two-sample comparison of means with cluster units: - n_per_arm ≈ 2 × (Z_{0.975} + Z_{0.8})^2 × σ^2 / δ^2 - With Z_{0.975}=1.96, Z_{0.8}=0.84, σ=2.5, δ=0.6 (pp), n ≈ 2 × (2.8^2) × (6.25) / (0.36) ≈ 272 store-weeks per arm. With 60 stores per arm over 6 weeks, power ≈ 0.8. - Estimator: Difference-in-Differences (DiD) with covariate adjustment (CUPED): - τ̂ = (Ȳ_T,post − Ȳ_T,pre) − (Ȳ_C,post − Ȳ_C,pre) - Regression with store fixed effects, week fixed effects, and controls (promotion intensity, average basket size, store traffic). - SEs: Cluster-robust at the store level. ## 7) Confounders & Data Gaps - Confounders: Retailer promotions, local events/weather, store staffing changes, seasonality, mix shifts to high-OOS categories. - Mitigations: Stratified randomization, pre-period matching, fixed effects, and explicit controls for promo intensity and category mix. - Data Gaps: Some stores lacked reliable inventory feeds; OOS reasons were noisily labeled. - Mitigations: Imputed inventory signal with historical pick-success priors, excluded low-quality feeds in a sensitivity analysis, and standardized reason codes using text classification on support notes. ## 8) Validation & Robustness - Assumptions: Checked parallel trends on the primary metric for 6 weeks pre-test; no significant divergence. - Placebos: No pre-period “effect” when pretending treatment started earlier. - Heterogeneity: Stronger effects in high-OOS stores; neutral in low-OOS stores. - Guardrails: Shopper time per order and app crash rate unchanged; contact rate decreased. - Sensitivity: Results consistent with and without low-quality inventory-feed stores; CUPED vs. raw DiD produced similar point estimates. - External validity: Ran a 2-week staged ramp to two new regions and replicated directionally similar effects before full rollout. ## 9) Results & Impact - Primary: Item-driven cancellations reduced from 6.1% → 5.0% (−1.1 pp; −18% relative; 95% CI: −0.7 to −1.5 pp; p < 0.01). - Secondary: Substitution acceptance +12% (from 52% → 58%); NPS +2.1 points; refund rate −0.4 pp. - Guardrails: Shopper time per order +0.1 min (ns); contact rate −6%. - Contribution Margin: +$0.42 per order. With 10M quarterly orders, incremental CM ≈ $4.2M per quarter (conservative, excludes potential LTV lift). Finance validated via weekly CM reconciliation. Formula example: CM_improvement ≈ (Refunds avoided per order × avg refund) + (Retention lift × expected GMV × margin) − (Feature opex per order). In our observed quarter, the refunds-avoided term dominated. ## 10) One Mistake and What I’d Do Differently Mistake: We underestimated the operational change-management for shoppers. Early adoption of the new substitution flow was only ~65%, dampening impact. What I’d change: Co-design with shopper champions, add in-app nudges/tooltips, set an adoption SLO (e.g., >85% within 2 weeks), and use a stepped-wedge rollout so each cohort can be trained and monitored before proceeding. ## Takeaways for Interviewers - Clear decision framing with business stakes - Proper experimental design and power - Explicit handling of confounders and data gaps - Robust validation and quantified financial impact - Reflective learning to improve the next iteration