Choose KPIs and prove impact with experiments

You’re on Expedia’s Search team launching a new hotel-ranking model for enterprise clients who care about profitable bookings, not clicks. a) Define a single primary KPI (give an exact formula) that aligns to client value (e.g., margin-adjusted bookings per search within a 7-day attribution window) and list at least three precise guardrail metrics with thresholds (e.g., cancellation rate ≤ X%, latency p95 ≤ Y ms, diversity entropy ≥ Z). b) Explain how you’d validate that your offline proxy metrics (e.g., NDCG@10 weighted by booking margin) correlate with the online KPI; specify the analysis design, acceptable correlation/elasticity ranges, and actions if misaligned. c) Design an A/B test: choose randomization unit (e.g., search session), traffic split, ramp strategy, MDE and sample-size calculation (state base rates/variance assumptions), pre-experiment checks (SRM), and variance reduction (e.g., CUPED/stratification). d) Describe how you’d detect and prevent metric gaming and novelty effects (e.g., clickbait, position churn) and set a rollback criterion. e) If bookings are flat but cancellations rise 15% in treatment, decide ship/rollback and justify with expected profit impact and confidence intervals. f) Estimate incremental gross profit per search with a 95% CI and describe how you’d separate incrementality from attribution. g) For paid channels, propose a multi-touch attribution approach (e.g., Shapley/Markov vs. regression vs. geo-experiments) and how you’d calibrate it against holdout/geo tests.