Demonstrate Amazon LP with deep follow-ups

# How to answer using STAR with quantification - Situation: Context, date, and scope. - Task: The specific outcome you owned with a target and deadline. - Action: What you did, how you did it, and why. Show mechanisms and trade-offs. - Result: Quantified impact. Net out benefits and costs. Quantification guardrails: - Use treatment minus control to estimate incremental impact when possible. - Net impact equals gross savings or revenue minus incremental costs and credits. - Include scope to make percentages tangible, for example, 2.6 percentage points on 2.1 million orders per quarter. --- ## 1) Customer Obsession — Reducing late delivery contacts via proactive interventions - Dates: Jan 2023 to May 2023 - Scope: US e-commerce orders across 3 categories, 2.1 million orders per quarter; CX metric baseline contact rate for late deliveries 9.3 percent; refunds 5.6 million dollars per quarter S — Situation - Late deliveries were the top driver of customer contact and refunds. Customer verbatims showed frustration with lack of proactive communication. T — Task - Reduce late-delivery contact rate by 20 percent before Mother’s Day while maintaining contribution margin. A — Action - Built a gradient-boosted model predicting delivery delays using features such as carrier, lane, weather, and historical SLA adherence. - Designed a policy: for top 10 percent highest risk orders, either upgrade shipping or send proactive ETA updates and a small goodwill credit when upgrade was not economical. - Ran a 50:50 A/B test across 2.1 million quarterly orders to measure incremental effect and cost; ramped from 10 percent to 50 percent to 100 percent over 6 weeks. - Built weekly monitoring and per-lane guardrails to suspend upgrades if carrier capacity tightened. R — Result - Late-delivery contact rate decreased by 28 percent relative, from 9.3 percent to 6.7 percent, equal to 54,600 fewer contacts per quarter. - Late deliveries dropped 12 percent relative; refunds decreased by 1.9 million dollars per quarter. - Incremental shipping upgrades cost 380 thousand dollars per quarter and goodwill credits 40 thousand dollars per quarter, net savings of approximately 1.48 million dollars per quarter. - NPS improved by 5.3 points in the treated cohort; repeat purchase rate increased by 1.1 percentage points within 60 days. Follow-ups - Metric moved and cost or trade-off: Contact rate down 2.6 percentage points; net savings 1.48 million dollars per quarter after 420 thousand dollars in costs. Trade-off: small margin hit on upgraded orders offset by reduced refunds and higher repeat purchase. - Disagree and commit and de-risk: PM wanted to upgrade 25 percent of orders at moderate risk. I disagreed and proposed a tiered policy backed by test power analysis. We disagreed and committed to my tiered approach with a 10 percent safety ramp and a lane-level kill switch. - Biggest mistake and prevention: Initially missed a regional feature interaction that over-upgraded one carrier lane, causing a one-week cost overrun of roughly 22 thousand dollars. Fixed with geospatial interaction terms and lane-level guardrails. Next time, I would include per-segment pre-checks and dry runs for top lanes. - Handling ambiguity or pushback: Finance pushed back on upgrade spend; CX pushed for broader coverage. I reconciled with a portfolio view – targeted high-risk orders only and weekly ROI reporting. - LPs also demonstrated: Dive Deep, Bias for Action, Frugality, Deliver Results. --- ## 2) Ownership — Standing up a self-serve experiment platform to accelerate learning - Dates: Jul 2022 to Nov 2022 - Scope: 35 product surfaces, 120 million monthly active users total exposure; 4 teams piloting; prior median analysis time 4 days S — Situation - Experimentation was bottlenecked. Analysts manually stitched logs; PMs waited days for readouts; weekly experiment throughput was 8. T — Task - Own delivery of a minimum viable self-serve experimentation platform by end of Nov 2022 that cuts analysis time by 50 percent and doubles weekly experiment throughput without compromising statistical rigor. A — Action - Drafted design for a standardized assignment service, metrics registry, and analysis pipelines using an open-source stats library with CUPED variance reduction. - Partnered with data engineering to define data contracts and event schemas. Prioritized an MVP: two-tailed tests, sequential monitoring with alpha spending, guardrails for power checks, and a pre-registered metrics catalog. - Shifted two sprints worth of my roadmap and secured one data engineer 50 percent allocation after aligning directors on ROI. - Ran a three-team pilot for 6 weeks, added user permissions and audit logs to address security concerns, and wrote playbooks and office hours for onboarding. R — Result - Median analysis time dropped from 4.0 days to 0.9 days, a 77 percent reduction. - Weekly experiment throughput increased from 8 to 22 within two months post-launch. - Estimated time savings of 1,200 analyst hours per year. Early wins unlocked 3.4 million dollars per year in incremental revenue by accelerating adoption of two features by 6 weeks. - Platform adoption reached 9 teams in 3 months; false positives reduced via pre-registration and power checks. Follow-ups - Metric moved and cost or trade-off: Throughput up 175 percent; analysis time down 77 percent. Trade-off: Deferred long-tail features like Bayesian bandits and heterogeneity reporting to hit date; used open-source components to control cost and accepted modest tech debt. - Disagree and commit and escalation: Data engineering wanted to block launch until a full rewrite of the event pipeline. I disagreed and proposed an adapter layer with validation. We escalated to the director-level review, committed to MVP with a 3-month sunset plan for the adapter. - Biggest mistake and prevention: Underinvested in training, leading to two early misconfigured experiments. I added templates, pre-launch checklists, and automatic power warnings. Next time I would schedule mandatory onboarding before granting write access. - Handling ambiguity or pushback: Security flagged PII risks. I added role-based access and metric-level anonymization to pass review. - LPs also demonstrated: Invent and Simplify, Insist on Highest Standards, Earn Trust, Are Right A Lot. --- ## 3) Missed Deadline — Holiday demand forecast delivered late and what I changed - Dates: Aug 2021 to Oct 2021 - Scope: Forecasting for Electronics category, 12 thousand SKUs, target launch by Oct 15 for a 12-week holiday horizon S — Situation - Leadership requested a new hierarchical demand forecast to reduce stockouts during holiday. Downstream buying decisions depended on the Oct 15 delivery. T — Task - Deliver a live forecast service and dashboard by Oct 15 with at least 10 percent MAPE improvement over baseline and 95 percent service reliability. A — Action - Chose a hybrid model (gradient boosting for short-term features plus top-down reconciliation). I added causal features for promotions and external signals. - I underestimated integration complexity and did not freeze features 4 weeks out. A late upstream schema change on Oct 5 invalidated our feature pipeline, and I kept adding features to chase accuracy. - On Oct 12, I escalated the risk, rebaselined with leadership, and pivoted to a staged delivery: baseline model on Oct 29 and hybrid by Nov 12. I also set up a shadow mode to validate without blocking decisions. R — Result - We missed the Oct 15 deadline and delivered baseline on Oct 29, 14 days late. Buying teams incurred approximately 450 thousand dollars in expedite and markdown costs during the gap. - By Nov 12, the hybrid model achieved 14 percent MAPE improvement and reduced stockouts by 18 percent relative, but the late start blunted peak benefits. - I implemented new mechanisms: feature freeze 4 weeks pre-launch, data contracts with upstream owners, stage gates, and a minimum viable baseline required by T minus 6 weeks. Follow-ups - Metric moved and cost or trade-off: Ultimately achieved 14 percent MAPE improvement and 18 percent stockout reduction post-launch. Trade-off: Accepted reduced scope at launch and incurred 450 thousand dollars in avoidable costs due to lateness. - Biggest mistake and prevention: Scope creep and failure to enforce a feature freeze. Prevention now includes a launch checklist, stage gates, and a red team review 30 days before launch. - Disagree and commit and de-risk: I wanted to delay launch for the higher-accuracy model; the GM insisted on something by Oct 29. I disagreed and committed to ship the baseline earlier with shadow validation and tight monitoring. - Handling ambiguity or pushback: Conflicting goals between buyers wanting conservative forecasts and finance wanting lean inventory. I built scenario ranges and communicated risk bands to align choices. - LPs also demonstrated: Ownership, Deliver Results, Dive Deep, Earn Trust. --- ## 4) Receiving Hard Negative Feedback — Making analyses actionable for business decision makers - Dates: Mar 2022 to Jun 2022 - Scope: Subscription retention and pricing analytics for 6 million active subscribers across 3 regions S — Situation - I presented an uplift modeling approach for retention campaigns. A senior PM told me the analysis was academic and not decision-ready, and an SVP said the deck did not answer so what. T — Task - Turn around trust by translating the work into clear decisions with dollarized impact within one month, and secure adoption across two pilot regions. A — Action - Reframed the narrative using an executive summary, decision trees, and clear if-then playbooks with thresholds. Focused on incremental revenue minus offer cost. - Ran a controlled 10 percent holdout test comparing uplift targeting against a simple recency-frequency heuristic the PM preferred. - Built a one-page dashboard highlighting next best action, expected net dollars per 1,000 users, and guardrails for budget caps; instituted weekly office hours for PMs. R — Result - Heuristic delivered 95 percent of uplift benefits at 30 percent of implementation complexity. We shipped the heuristic first and scheduled uplift for phase 2. - Net retention revenue increased by 2.8 million dollars per year across two regions with an average 1.2 percentage point improvement in 90-day retention among targeted users. - Stakeholder survey moved from 2.6 to 4.4 out of 5 on decision readiness. The approach was adopted by 9 teams within 3 months. Follow-ups - Metric moved and cost or trade-off: Net retention revenue plus 2.8 million dollars per year; average retention plus 1.2 percentage points. Trade-off: Deferred complex modeling in favor of faster time-to-value; accepted slightly lower lift for faster adoption. - Disagree and commit and de-risk: I preferred uplift modeling; PM preferred heuristic. We ran an A/B test, aligned on heuristic first, and committed to revisit uplift under a clearer ROI threshold. De-risked with a 10 percent holdout and budget guardrails. - Biggest mistake and prevention: I led with ROC-AUC and model internals instead of business impact. Now I start with the decision, quantify dollars per 1,000 users, and keep a one-page memo with Pyramid Principle. - Handling ambiguity or pushback: Multiple PMs had conflicting KPI preferences. I standardized on net revenue as north star with secondary guardrails on churn and LTV; socialized this in a metrics doc. - LPs also demonstrated: Earn Trust, Learn and Be Curious, Insist on Highest Standards, Customer Obsession. --- Validation notes for experimentation - Power analysis before committing targets to avoid false negatives and overspending. - Pre-register primary metrics and guardrail metrics; monitor for peeking via alpha spending or fixed horizon. - Report net impact by subtracting incremental costs; include confidence intervals or MDE when relevant.