# Model Answer: A High-Stakes Data Decision, End to End This is a behavioral question, so the goal is a **structured, honest, quantified story** that shows both statistical rigor and cross-functional judgment. Below is (A) a reusable framework for answering it, (B) a fully worked example you can use as a template, and (C) what separates a strong answer from a weak one. --- ## A) How to Structure the Answer Use a data-science adaptation of **STAR**, weighted toward Action and Result: - **Situation (10–15s)** — the business context and why the decision was high-stakes. - **Task (10–15s)** — the specific decision you owned and the uncertainty involved. - **Action (60%)** — hypotheses, metrics, data, design, confounders, validation. This is where you demonstrate rigor and judgment. - **Result (25%)** — quantified impact with uncertainty, the decision made, and one honest lesson. Cover the ten elements the prompt lists, but narrate them as a story, not a checklist. Lead with the decision and the stakes so the interviewer immediately understands why your work mattered. --- ## B) Worked Example: Reducing Out-of-Stock Cancellations in a Same-Day Grocery Marketplace ### 1) Decision We had to decide whether to launch a **real-time substitution + inventory-quality feature** to cut order cancellations caused by out-of-stock (OOS) items, and if so, to whom: (a) all stores, (b) only high-OOS stores, or (c) hold and iterate. **Why it mattered:** OOS-driven cancellations triggered refunds, shopper-support contacts, and customer churn — hitting contribution margin (CM) and customer LTV directly. A bad launch could also slow shoppers down, harming throughput. ### 2) Hypotheses - **H1 (primary):** Real-time, model-driven substitutions plus better inventory signals reduce item-driven cancellations. - **H2 (alternative / mechanism):** The feature increases *substitution acceptance* rather than reducing cancellations — i.e., it shifts behavior but doesn't move the headline metric. - **H3 (cost concern):** Any cancellation gain is offset by **slower shopper time per order**, hurting throughput. - **H0 (null):** No effect beyond noise. Framing a real alternative (H2/H3) matters: it forces guardrails and prevents declaring victory on a metric that merely moved sideways. ### 3) Success Metrics - **Primary:** Item-driven cancellation rate (OOS-attributed cancellations per order). - **Secondary:** Substitution acceptance rate, refund rate, customer CSAT/NPS, per-order contribution margin. - **Guardrails (must not regress):** Shopper time per order, customer support contact rate, app crash/error rate. One primary metric keeps the decision unambiguous; guardrails encode the alternative hypotheses so we can't win on cancellations while quietly hurting throughput or stability. ### 4) Stakeholders - **Product & Engineering** — feature design, rollout, and the experimentation platform. - **Operations (Shopper Ops, Retail Ops)** — shopper training and retailer coordination; they cared about throughput and shopper experience. - **Finance** — owned the margin model and validated the dollar impact. - **CX / Support** — cared about contact volume and resolution time. I aligned on the primary metric and guardrails with all four *before* launch so the ship/no-ship rule was agreed in advance, not litigated after the fact. ### 5) Data Sources & Trustworthiness - Order and line-item event logs (add-to-cart, pick, substitute, refund, cancel) with reason codes. - Shopper app telemetry (time per item, substitution offers/accepts). - Retailer inventory feeds (where available) + historical sell-through. - Catalog/store metadata (availability flags, store traffic, hours) and promotion/pricing tables. - Experiment assignment and exposure logs. **Why trustworthy enough:** I deduplicated line items, reconciled event-clock skew across systems, standardized noisy OOS reason codes into a controlled vocabulary, and built a **store-level OOS-quality score** from historical pick success so I could quantify (not assume) each store's data reliability. Stores with unreliable feeds were flagged for a sensitivity analysis rather than silently trusted. ### 6) Method: Cluster-Randomized Geo-Experiment - **Design:** Cluster-randomized A/B test with the **store as the unit of randomization**, run for **6 weeks**. Randomizing at the store (not the order or shopper) avoids contamination — a shopper or retailer can't be half in treatment — and matches how the feature is actually deployed. - **Stratified assignment:** Within each *retailer × region* stratum, randomly assign stores to Treatment or Control, balancing retailer mix and regional seasonality. - **Power / sample size:** Baseline cancellation rate ≈ 6.0%, **single-week within-store SD** σ₁ ≈ 2.5 pp (the week-to-week variability for a given store around its 6-week mean). Targeting a **10% relative reduction** (δ ≈ 0.6 pp): - Each store's *6-week average* has SD $\sigma_{\bar{w}} = \sigma_1/\sqrt{6} \approx 2.5/\sqrt{6} \approx 1.02$ pp, because the six weekly observations within a store are approximately exchangeable after conditioning on the store and week fixed effects. - The formula for two independent groups of stores, powered on the store-level 6-week average: $$ n \approx 2\,(z_{0.975}+z_{0.80})^2\,\frac{\sigma_{\bar{w}}^2}{\delta^2} = 2\,(1.96+0.84)^2\,\frac{1.02^2}{0.6^2} \approx 2 \times 7.84 \times \frac{1.04}{0.36} \approx 45 $$ store-units per arm for ~80% power at α = 0.05. **60 stores per arm comfortably exceeds 45**, providing >85% power on this MDE. - *Contrast with the naive formula:* plugging the single-week SD (σ₁ = 2.5 pp) into the same formula without accounting for averaging would yield n ≈ 272 — but that would be the requirement if the outcome were a single randomly chosen week per store. Because the outcome is the **6-week average per store**, the relevant variance is σ₁²/6, which is what determines power here. - I did **not** treat the 360 store-weeks as 360 independent units — clustering is real, and I powered on stores and used cluster-robust SEs (below) to keep Type-I error calibrated. - **Estimator:** Difference-in-Differences with CUPED-style covariate adjustment: $$ \hat{\tau} = (\bar{Y}_{T,\text{post}} - \bar{Y}_{T,\text{pre}}) - (\bar{Y}_{C,\text{post}} - \bar{Y}_{C,\text{pre}}) $$ fit as a regression with **store and week fixed effects** plus controls (promo intensity, average basket size, store traffic). Pre-period covariate adjustment (CUPED) reduced variance and tightened the CI. - **Inference:** **Cluster-robust standard errors at the store level** so dependence within a store over weeks doesn't inflate significance. ### 7) Confounders & Data Gaps - **Confounders:** retailer promotions, local events/weather, staffing changes, seasonality, and basket mix-shift toward high-OOS categories. - *Mitigations:* stratified randomization, store & week fixed effects, explicit controls for promo intensity and category mix, and DiD to net out time-invariant store differences. - **Data gaps:** some stores lacked reliable inventory feeds; OOS reason labels were noisy. - *Mitigations:* imputed a missing inventory signal from historical pick-success priors, standardized reason codes via a text classifier on support notes, and **pre-registered a sensitivity analysis** excluding low-quality-feed stores. ### 8) Validation & Robustness - **Parallel-trends check:** verified treatment and control tracked together on the primary metric across the 6-week pre-period — the key DiD assumption. - **Balance check:** confirmed covariate balance across arms post-randomization. - **Placebo test:** assigned a fake "treatment start" in the pre-period; no spurious effect appeared. - **Heterogeneity:** effect concentrated in high-OOS stores, near-zero in low-OOS stores — consistent with the mechanism and informing a targeted rollout. - **Guardrails held:** shopper time per order and crash rate unchanged; contact rate fell. - **Sensitivity:** results stable with and without low-quality-feed stores; CUPED and raw DiD gave similar point estimates. - **External validity:** a 2-week staged ramp into two new regions reproduced directionally similar effects before full rollout. ### 9) Results & Impact - **Primary:** item-driven cancellations fell **6.1% → 5.0%** (−1.1 pp, **−18% relative**; 95% CI roughly −0.7 to −1.5 pp; p < 0.01). - **Secondary:** substitution acceptance +6 pp (52% → 58%); refund rate −0.4 pp; NPS +2 points. - **Guardrails:** shopper time per order +0.1 min (not significant); contact rate −6%. - **Contribution margin:** ≈ **+$0.42 per order**, dominated by avoided refunds. At ~10M quarterly orders that is **≈ $4.2M incremental CM per quarter** — a deliberately *conservative* figure that excludes the harder-to-attribute retention/LTV lift. Finance independently validated the per-order CM via weekly reconciliation. I framed the CM as a directional model, not a precise claim: $$ \Delta\text{CM} \approx (\text{refunds avoided} \times \text{avg refund}) + (\text{retention lift} \times \text{expected GMV} \times \text{margin}) - \text{feature opex per order} $$ and was explicit that the retention term was the **least certain** input — so I reported the refunds-avoided portion as the defensible floor and the retention lift as upside. **Decision made:** launch to **all stores** but with adoption monitoring, because the high-OOS gain was real and guardrails held; the low-OOS neutral result meant little downside to a broad rollout. ### 10) One Mistake & What I'd Do Differently **Mistake:** I underweighted **change management for shoppers**. Early adoption of the new substitution flow was only ~65%, which diluted the measured effect — the feature worked better than the intent-to-treat estimate suggested. **What I'd change:** co-design with shopper champions, add in-app nudges/tooltips, set an explicit **adoption SLO** (e.g., >85% within 2 weeks), and use a **stepped-wedge rollout** so each cohort is trained and monitored before the next turns on — improving both adoption and the cleanliness of the estimate. --- ## Addressing Likely Follow-ups - **Why store-level randomization vs. user-level?** The feature changes the *shopping* experience and depends on store inventory, so treatment can't be isolated to a single user — a shopper fulfilling orders in a store would be partly exposed regardless. User-level randomization would cause **interference (SUTVA violation)** and contaminate control. The cost is fewer independent units and lower power, which is exactly why I powered on stores and used cluster-robust SEs. User-level would only be appropriate for a purely app-side, per-user UI change with no shared resource. - **If the result had been null:** I'd distinguish "no effect" from "underpowered." I'd check the CI width against the minimum *business-relevant* effect — if the CI excluded a meaningful lift, kill it; if it was wide and adoption was low (as here), the honest read is "inconclusive due to dilution," so iterate on adoption and re-run rather than ship or kill outright. - **Guardrail regressed while primary improved:** revert to the pre-agreed rule. Compare the guardrail breach to its pre-set tolerance and translate both into the *same currency* (CM / customer harm). A tiny, non-significant shopper-time increase against a large cancellation win is a ship; a real CSAT or crash-rate regression is a no-ship pending a fix — and I'd never decide this unilaterally after the fact. - **Pushback from Finance on the dollar impact:** treat it as a feature, not a fight. I co-owned the CM model with Finance up front, reported the conservative refunds-avoided floor separately from speculative LTV upside, and offered to re-run the reconciliation on their preferred cohort. Aligning on the model *before* the readout prevents the result from being relitigated. --- ## C) What Separates a Strong Answer From a Weak One | Dimension | Strong | Weak | |---|---|---| | Decision | Clear, consequential, tied to a business metric | Vague "analysis I did" with no decision at stake | | Hypotheses | Falsifiable primary + real alternative | Only a hoped-for outcome, no null | | Metrics | Primary + secondary + guardrails | One metric, no guardrails | | Causal rigor | Right unit, real power argument, correct clustering | "We saw the number go up" | | Confounders | Named threats + concrete mitigations | None mentioned | | Validation | Assumption checks, placebos, sensitivity, replication | A single p-value | | Impact | Effect size + CI + honest dollar range | "It helped a lot" | | Reflection | A genuine mistake + concrete fix | "I'd do everything the same" | **The single most common failure** in DS behavioral rounds is unquantified impact and no causal design. Lead with the decision and stakes, spend your time on *why* you chose the design, and close with honest numbers and one real lesson.