Expedia Search: Profit-Focused Ranking Launch (Onsite)

Context

You’re on the Search team launching a new hotel-ranking model for enterprise clients who care about profitable bookings, not clicks. Assume attribution uses a 7-day booking window from the search event. Cancellations typically occur within 30 days, and you can estimate an expected show-up probability at booking time.

Tasks

(a) Define a single primary KPI with an exact formula that aligns to client value (e.g., margin-adjusted bookings per search within a 7-day attribution window). List at least three guardrail metrics with explicit numeric thresholds (e.g., cancellation rate ≤ X%, latency p95 ≤ Y ms, diversity entropy ≥ Z).

(b) Explain how you would validate that offline proxy metrics (e.g., NDCG@10 weighted by booking margin) correlate with the online KPI. Specify the analysis design, acceptable correlation/elasticity ranges, and what you would do if they are misaligned.

(c) Design an A/B test: choose the randomization unit (e.g., search session), traffic split, ramp strategy, MDE and sample-size calculation (state base rates/variance assumptions), pre-experiment checks (e.g., SRM), and variance reduction (e.g., CUPED/stratification).

(d) Describe how you would detect and prevent metric gaming and novelty effects (e.g., clickbait, position churn) and set a rollback criterion.

(e) If bookings are flat but cancellations rise 15% in treatment, decide ship or rollback and justify with expected profit impact and confidence intervals.

(f) Estimate incremental gross profit per search with a 95% CI and describe how you would separate incrementality from attribution.

(g) For paid channels, propose a multi-touch attribution approach (e.g., Shapley/Markov vs. regression vs. geo-experiments) and how you would calibrate it against holdout/geo tests.