Demonstrate problem-solving under resistance

# Example STAR Answer (Data Scientist) — Launching a New Demand Forecasting System Under Resistance Below is a comprehensive, teaching-oriented example. It shows how to tie business impact to experimentation, risk management, and organization-wide adoption. ## Situation - Context: An e-commerce marketplace suffered frequent stockouts and overstock during seasonal peaks. Category managers relied on manual heuristics; the rule-based demand forecasts had high error on promotions and new items. - Business target (12-week deadline before peak season): - Reduce stockouts by 20% on treated SKUs. - Improve forecast accuracy (MAPE) by 15% relative. - Cut manual overrides by 50%. - Do no harm to gross margin; keep compute cost per 1k forecasts flat or lower. Assumptions to make it concrete: - ~300k SKUs across 15 category orgs; nightly batch replenishment writes purchase orders. - Existing MAPE ~28%; manual override rate ~40% of items. ## Task - Deliver an end-to-end forecasting upgrade (feature engineering, model, validation, deployment, guardrails) and drive adoption across planning teams and the replenishment platform. - Define success metrics and risk thresholds acceptable to operations and finance. Primary metrics and formulas: - MAPE: MAPE = (1/n) Σ |(y − ŷ)/y|. - Bias: mean((ŷ − y)/y). - Business: stockout rate, GMV/units sold, margin, manual override rate, inference cost per 1k forecasts. ## Obstacles - Resistance from planners: fear of risky automated changes leading to stockouts during peak. - Platform/infra pushback: concern over higher inference cost and potential latency spikes. - Data quality: promo flags and price history had late-arriving updates and missing values. - Cold start risk: new SKUs and highly seasonal items. - Governance: replenishment job was owned by another org; changes required code-owner buy-in. ## Actions 1) Diagnosis and baselines - Explored 2 years of demand, price, promo, cannibalization signals, calendar features, and competitor availability proxies. - Identified root causes: promo uplift under-modeled; bias during peak season; heuristics ignored cross-item cannibalization. - Built a strong baseline (seasonal naive + Prophet) and a candidate XGBoost model with hierarchical reconciliation to category totals. 2) Validation strategy and risk guardrails - Time-series cross-validation (rolling windows) to prevent leakage; backtested on last 6 seasonal cycles. - Shadow mode: ran new forecasts in parallel for 4 weeks; compared MAPE, bias, and simulated service level without affecting orders. - Defined launch SLAs: MAPE ≤ 22% and bias within ±5% for each category-week; if violated, auto-fallback to baseline for that slice. - Three-tier risk gating by SKU: - Tier A (low risk, high-volume): full automation. - Tier B (medium): automated but capped uplift vs. baseline. - Tier C (high risk/new): decision support only, planner confirmation required. 3) Experimentation and decisions - A/B test at SKU×region level (10% holdout per category) for 6 weeks; stratified by velocity and promo intensity. - CUPED variance reduction using pre-period demand to tighten confidence intervals. - Added cost controls: batch nightly inference; quantized model and feature caching; reduced per-1k forecasts compute by ~40%. - Cold-start fix: Bayesian shrinkage to category priors plus similar-item features; fallback to baseline when uncertainty high. 4) Socialization, escalation, and alignment - Weekly forum with planners, finance, and platform leads: published dashboards showing MAPE, bias, stockouts, and simulated P&L. - Pre-mortem with dissenters: documented failure modes and specific kill-switches; secured sign-off on ramp criteria. - Escalation: presented a PR/FAQ and risk-return analysis to directors of operations and platform to secure deployment windows and resourcing. 5) Adoption plan and instrumentation - Feature flags per org: staged ramp 0% → 10% → 50% → 90% based on SLAs. - Training and playbooks: short videos, office hours, “how to debug forecasts” guides. - Tooling defaults: in the planning UI, new forecasts became the default view with a one-click revert to baseline. - Code-owner adoption: moved replenishment job to shared ownership; published a versioned forecast library on the internal package index; created migration PRs for 8 repos. - Usage telemetry: logged forecast API calls by org, planner WAU/DAU, override counts, and support tickets by category. ## Results - Accuracy and operations - MAPE improved from 28.0% to 18.0% on treated SKUs (a 36% relative improvement). - Bias tightened from +7% to +2% in peak weeks. - Stockout rate decreased by 22% on treated cohorts; manual overrides fell by 63%. - GMV increased by 3.1% on treated SKUs; margin impact +$12.4M annualized (finance-validated, difference-in-differences with CUPED). - Compute cost per 1k forecasts decreased by 40% via quantization and batching. - Adoption and verification - Feature-flag exposure: ramped to 92% across 14/15 orgs within 8 weeks; last org remained at 60% pending a seasonal event. - Active users: planner WAU rose from 40 to 230; DAU/WAU stabilized at ~62% with median session 14 minutes. API calls per org increased 4×. - Code-owner adoption: 8 repositories migrated to the shared forecast library; 15 PRs merged with 5 distinct org code-owners co-signing; the replenishment pipeline OWNER file updated to joint ownership. - Support tickets: forecast-related tickets dropped 58% (from 86/month to 36/month); median TTR improved from 2.1 days to 0.9 days. - Statistical confidence (example): - A/B uplift in stockout rate: −2.8 pp (95% CI: −3.4, −2.2) using cluster-robust SEs at category level. - MAPE improvement consistently met SLA across 13/15 orgs in backtests and live. ## Handling Dissent and Trade-offs - Planners’ concern about peak risk: instituted tiered rollouts, per-slice SLAs, and hard caps on forecast deltas during the first two peak weeks. - Infra team’s cost concerns: avoided online scoring; used nightly batches, feature stores, and model quantization; published a cost telemetry dashboard. - Category with high newness (toys) resisted adoption: we added explicit cold-start uncertainty flags; kept them at decision support (Tier C) until post-peak, then ramped after cold-start performance passed thresholds. Trade-offs: - Accepted slightly worse accuracy on long-tail SKUs to harden against over-ordering; prioritized high-volume items for gains. - Chose interpretability aids (SHAP, monotonic constraints) over marginal accuracy to build trust. ## What I’d Do Differently - Invest earlier in a policy simulator to estimate inventory and P&L effects before live A/B, reducing ramp time. - Formalize data contracts with upstream promo/price teams to prevent late-arriving fields and schema drift. - Pre-plan enablement with a dedicated change manager per org; adoption accelerated notably where we co-ran training with line managers. - Expand guardrails to include service-level targets per fulfillment node, not just category-week averages. ## Notes for Interview Delivery - Keep it tight: 2–3 minutes per STAR section. - Lead with business impact and risk mitigation; show you controlled the blast radius. - Cite 3–4 concrete adoption metrics (flag exposure %, WAU by org, code-owner PRs, ticket trends) and one cost/latency metric. - If you lack real numbers, use directional results plus the exact methods you would use to verify adoption.