Describe a decision with incomplete information

# How to Craft a Strong Answer Use a clear structure that demonstrates judgment under uncertainty: - Situation/Task: Brief context and stakes. - Alternatives: Options you considered (including "do nothing"). - Assumptions & Risks: What you believed; what could fail. - Rapid Evidence: Just-enough data to de-risk (offline/online checks, quick math). - Decision & Guardrails: The choice, why it was reversible or not, and controls. - Outcome: Concrete results. - Learnings: What you’d do differently and how you mitigated risk. Below is a model example tailored to a Machine Learning Engineer scenario. ## Example Answer (MLE Context) 1) Context - I led the launch of a new ranking model for the homepage feed two weeks before a major seasonal traffic spike. Offline metrics (NDCG@10, AUC) showed improvements, but we lacked time and traffic to run a full-powered A/B before the event. Constraints: strict p95 latency ≤ 200 ms, limited feature store backfills for some cohorts, and an infra freeze date. 2) Options Considered - Option A: Delay the launch until after the event; continue offline validation. - Option B: Shadow mode only (log predictions without serving) through the event; launch later. - Option C: Canary to a small percentage with tight guardrails and a fast rollback, combined with short shadow-mode validation. 3) Assumptions and Risks - Assumptions: (a) Offline-to-online correlation is positive; (b) latency headroom is sufficient with dynamic batching; (c) limited missing-value imputation won’t harm key cohorts. - Risks: (a) CTR or conversion could drop; (b) p95 latency SLO could be breached; (c) novelty effects or distribution shift during the event; (d) long-tail fairness (low-traffic locales) issues. 4) Just-Enough Data (Fast Evidence) - Log replay performance: Replayed 24 hours of traffic to measure inference latency and memory. Result: model p95 inference 45 ms; pipeline p95 projected 185 ms (SLO 200 ms OK). - Shadow validation (48 hours): Compared new vs. old ranker scores on the same requests; no regressions on key segments. Checked for leakage and feature parity across top 10 features. - Back-of-the-envelope impact: For a 10% canary with ~1,000,000 impressions/day, baseline CTR ≈ 8%. Offline predicted relative uplift ≈ 1.5% → expected CTR ≈ 8.12%. - Extra clicks/day ≈ 1,000,000 × (0.0812 − 0.08) = 1,200. - If CVR ≈ 5% and AOV ≈ $45, incremental revenue/day ≈ 1,200 × 0.05 × 45 = $2,700. - Worst-case guardrail for risk: cap downside if CTR drops by 0.5% relative → loss bounded by ≈ 1,000,000 × 0.08 × 0.005 × 0.05 × 45 ≈ $900/day at 10%. - A/B power check (is a full test feasible?): To detect an absolute CTR lift δ = 0.2% (0.002) around p = 0.08, a rough sample size per arm is n ≈ 16 p(1−p)/δ² ≈ 16×0.0736/0.000004 ≈ 294,400 impressions/arm. With the time constraint, we could get this within a day at 10% canary, making a small, reversible online read feasible. 5) Decision and Guardrails - I chose Option C (canary) because it was reversible and bounded risk. We: - Launched to 10% traffic with a kill switch and auto-rollback if any guardrail breached for >15 minutes. - Guardrails: CTR drop > 0.5% relative, conversion drop > 0.3% relative, p95 latency > 200 ms, error rate > 0.3%. - Exclusions: High-value segments (e.g., wholesale accounts) initially excluded to limit downside. - Monitoring: Real-time dashboards with cohort cuts (new vs. returning, locale, device), plus p95/p99 latency panels. - Technical controls: Fallback to previous model if inference > 150 ms mid-pipeline; circuit-breaker on feature store timeouts. 6) Outcome - After 24 hours, canary showed: CTR +1.2% relative, conversion +0.3% relative, p95 latency 185 ms, no increase in errors. No fairness alerts across low-traffic locales. We ramped to 50% next day and 100% by day three. - Over the event week, incremental revenue estimated at ~+$18k vs. baseline for the canary-then-ramp period, with stable latency and no on-call incidents. 7) Retrospective and Mitigations - What I’d do differently: Instrument long-term retention proxies earlier; set up offline→online correlation tracking to avoid last-minute analysis; pre-stage logs for rare cohorts to tighten confidence intervals. - How I mitigated downside: Treated it as a reversible decision with strict guardrails, staged rollouts, cohort-level monitors, and a tested rollback path. We also documented assumptions and thresholds so on-call could act without waiting for me. ## Why This Works (Transferable Principles) - Reversible vs. irreversible: Favor a small, reversible decision with guardrails when time is short. - Quantify uncertainty: Use quick math to bound upside/downside and justify canary size. - Corroborate with multiple weak signals: Offline metrics + shadow mode + limited canary beats waiting for perfect data. - Guardrails and observability: Define trigger thresholds, automate rollback, and monitor by cohort (avoid Simpson’s paradox). - Validate constraints: Check p95/p99 latency, error budgets, and feature parity to prevent non-metric failures. ## Pitfalls and Edge Cases to Call Out - Offline-to-online mismatch: Offline metrics can overstate gains; test with shadow and small online exposure. - Instrumentation gaps: Missing events can hide regressions; verify tracking before rollout. - Distribution shift: Big events change behavior; compare like-for-like cohorts and time windows. - Long-tail cohorts: Ensure low-traffic locales/devices aren’t regressing; use stratified views. Use this template to plug in your own project details. Keep the story tight (2–3 minutes spoken), emphasize your judgment, and quantify both risk and outcome.