Demonstrate leadership under strict rules

# STAR Answer Example — Delivering Results Under a Non‑Negotiable Experimentation Policy Situation (Jan 10–Mar 31, 2023) - Context: I led experimentation for the Product Detail Page (PDP) of a large e‑commerce marketplace. The team proposed a persistent “Buy Now” button on mobile to lift add‑to‑cart (ATC) and conversion ahead of quarter end. - Tension: Product and Marketing wanted to ship after five days of promising early results. Company policy mandated strict experimentation standards before any broad rollout. Task - Measurable target I owned: Deliver a +30 bps absolute lift in PDP add‑to‑cart rate by 2023‑03‑31 without degrading critical guardrails (refund rate, page latency, customer contacts). - Stakeholders to influence: - Product Manager (PDP) - Engineering Manager (Web/Mobile) - Marketing Director (Mobile Growth) - Experimentation Program Manager (central platform) - Privacy/Compliance Officer - VP, Commerce (exec sponsor) Action - The non‑negotiable rule/policy and why it existed - Experimentation Governance Policy (EGP), effective company‑wide: 1) Minimum runtime: 14 consecutive days spanning two full weekly cycles to control for day‑of‑week seasonality. 2) Pre‑registered primary metric and Minimum Detectable Effect (MDE) with ≥95% power; no p‑hacking or unplanned early stopping. 3) Guardrails must not breach: refund rate, CS contacts, and p95 page load time; staged rollout only after significance and guardrail checks. - Rationale: Prevent false positives, protect customers from regressions, ensure long‑term trust and scientific rigor. - Timeline and key steps (with dates) - 2023‑01‑10: Kickoff; clarified target and constraints. - 2023‑01‑12: Wrote a 2‑page plan: hypotheses, pre‑registered metrics, MDE, power calc, guardrails, ramp plan; socialized with PM, Eng, Marketing, Experimentation lead, Privacy. - 2023‑01‑17: Final design sign‑off; variant behind a feature flag; CUPED enabled to reduce variance; holdout 10% persistent control for post‑ramp monitoring. - 2023‑01‑23: Launch A/B at 50/50 to accelerate learning while honoring policy. - 2023‑01‑27 (Day 5): Early uplift looked large (+1.2% ATC). Team asked to ship. I held the line: no early stopping per EGP; shared a one‑pager explaining peeking risk with a simulation showing a ~20–30% inflated Type I error when stopping on day‑5 spikes. - 2023‑02‑05: Completed 14‑day run; results significant; no guardrail breaches. - 2023‑02‑07–02‑10: Staged ramp to 100% with 48‑hour guardrail monitoring; post‑ramp persistent control validated stability. - Through 2023‑03‑31: Weekly readouts; instrumentation and latency tuning. - Risk, ethics, and safeguards - Risks: False positives from seasonality/novelty effects; customer harm via accidental purchases; degraded performance (page speed); analyst bias (p‑hacking); privacy violations. - Safeguards: Pre‑registration, CUPED to improve sensitivity without peeking, guardrail thresholds (refunds, CS contacts, latency), staged rollout, privacy‑safe aggregates only. - Alternatives considered and rejected (and why) - Ship at day 5 based on interim sig: Rejected; violates no‑peeking and increases false‑positive risk. - Cherry‑pick segments that looked best to claim success: Rejected; not pre‑registered; violates analysis integrity. - Shorten runtime to 7 days: Rejected; would not cover full weekly cycles; higher variance. - Export raw user‑level logs to personal machine to speed analysis: Rejected; privacy/compliance risk; not necessary with platform aggregates. Result (quantified) - Primary metric: +0.58 percentage points absolute ATC lift (95% CI: +0.35 to +0.82) at 14 days. - Business impact: Annualized +$3.4M incremental gross merchandise value (conservative LTV model, holdout‑validated). - Guardrails: No significant increase in refund rate; p95 page load +2.1% initially — we optimized image assets to bring it to +0.4% within a week. - Risk avoided: An unreviewed mobile sub‑variant increased accidental single‑item purchases by ~8%; guardrails caught it during staged ramp. We added a confirmation interstitial on edge cases before 100% rollout. - Stakeholder alignment: PM/Eng/Marketing agreed to adopt the same pre‑reg + guardrail template for all high‑impact PDP tests going forward. Reflection — What I would change next time - Pre‑alignment: Conduct a 30‑minute expectation‑setting session at kickoff to agree on minimum runtime, guardrails, and the communication plan to reduce pressure mid‑run. - Faster learning within the rules: Standardize CUPED and variance‑reduction features; instrument a sequential monitoring approach only if and when the policy is updated to allow alpha‑spending (while keeping error control explicit). - Mechanisms: Automate a weekly “green/yellow/red” dashboard tied to pre‑registered metrics and guardrails so stakeholders see progress without asking for interim peeks. --- Follow‑Up: Conflicting directives mid‑project and two weeks of negative trends — how I realign and course‑correct without violating the rule Scenario: Two key stakeholders (PM wants to continue ramp to hit feature adoption; Marketing wants a rollback due to campaign KPIs) give conflicting directives. For two consecutive weeks, primary or guardrail metrics trend down. Step‑by‑step plan (48–72 hours) 1) Create a one‑page alignment brief (same day) - Top: Restate the non‑negotiable policy (runtime, pre‑reg, guardrails, no peeking/stop without criteria). - Current state: Last 14 days of metrics with CIs; trend lines; which guardrails tripped and when. - Single north‑star metric and guardrails; explicitly list decision criteria (e.g., stop‑loss if refund rate > +X% week‑over‑week for 7 days). 2) Convene both stakeholders + Experimentation lead (within 24 hours) - Facilitate a decision using facts: Show the risk of violating the policy and the cost of a wrong call. - Propose an options matrix that all comply with the rule: - Option A: Freeze ramp at current exposure; continue to full pre‑registered runtime; add monitoring deep‑dive. - Option B: Rule‑compliant rollback trigger if guardrail breaches persist for N days (pre‑defined stop‑loss), then design a follow‑up experiment. - Option C: Narrow‑scope variant (e.g., mobile‑only off or exclude high‑risk cohorts) launched as a new experiment with its own pre‑reg plan. 3) Immediate course correction (no rule violations) - If two consecutive weeks show statistically meaningful degradation in a guardrail, activate the pre‑defined rollback to control or to a safer variant (per policy’s stop‑loss). - Run a rapid, policy‑compliant root‑cause analysis: - Slice by device, traffic source, page speed buckets; check instrumentation; ensure sample‑ratio mismatch (SRM) not present. - Use variance reduction (e.g., CUPED) to sharpen estimates; do not alter pre‑registered metrics. 4) Clarify single ownership and cadence - Identify the DRI (usually PM with DS and Exp Program as approvers) and set a twice‑weekly readout until metrics stabilize. - Document the decision and rationale; “disagree and commit” once chosen. 5) Escalation path (if deadlocked) - If conflict persists after the meeting, escalate with the one‑pager to the next‑level leader for a tie‑break within 24 hours, keeping the experiment within policy until a decision is made. Why this works - It protects customers and long‑term trust (no policy breach), restores a single source of truth, and uses pre‑agreed stop‑loss mechanisms to act decisively while maintaining statistical and ethical rigor.