Choosing Between Two Travel Search Algorithms: North Star Metric and Launch Criteria

You are a product manager at Agoda. The team has two versions of the search algorithm, and you need to decide which one should serve travelers. Define the north star KPI you would compare them on, name three metrics through which you would measure that KPI, and state the success flags that would tell you one algorithm has won. ```hint Follow the search to the booking A search algorithm is only one step on a traveler's path from entering a destination to completing a booking. Anchor the north star in the outcome of that path, then pick metrics that each explain a different way an algorithm could move it. ``` ### Clarifying Questions - Which search are we comparing: the suggestions offered while the traveler types a destination, or the ranking of properties on the results page? And for which product, such as hotels, homes or flights? - Can the two algorithms be tested side by side on live traffic with random assignment, and is one of them the version running today? - Is the goal more bookings only, or also the value of those bookings, given that a ranking change can shift which properties and price levels travelers choose? - Are there limits on response time or serving cost that either algorithm must stay within? ### What a Strong Answer Covers - Scoping which search and which part of the traveler journey are in play, and judging the algorithms on outcomes rather than on how they work internally. - A precisely defined north star tied to searches that end in bookings, including its unit, time window and denominator. - Three supporting metrics that each explain a distinct way the north star can move (relevance of what ranks at the top, friction during the search session, whether clicks turn into bookings), plus guardrails such as latency, cancellations and revenue per booking. - Success flags fixed before the test: thresholds, statistical confidence, guardrail tolerances, segment consistency, and a rule for when metrics disagree. - An experiment design suited to travel: randomization unit, duration across full weeks, bookings that happen days after the search, and novelty effects. ### Follow-up Questions - Algorithm B lifts bookings per searcher but is noticeably slower to load the results page. How do you decide whether to ship it? - Travelers often search several times over several days, sometimes on different devices, before booking. How does that change your metric definitions and test design? - B wins overall but loses for new users in one important market. What would you do?