Design Restaurant Search Ranking: Features, Labels and Marketplace A/B Tests

Design the ranking system behind restaurant search in a food-delivery app: a customer types a query, such as a cuisine, a dish or a restaurant name, and the app returns a ranked list of restaurants that can deliver to them. The discussion went deep on two topics: which features and labels you would use to train the ranker, and how you would design the online A/B test, including the case where a large ranking change shifts enough orders to strain the shared courier supply, so that a simple comparison between treatment and control is biased. ### Clarifying Questions - What is the objective: more orders per search, order value, long-term retention, or a mix with delivery-quality guardrails? - What kinds of queries arrive (restaurant names, cuisines, dishes, browsing with an empty query), and how many restaurants are eligible for a typical address? - What latency budget does the search response have? - Are sponsored placements mixed into the results? - What is logged today: impressions with positions, clicks, orders, ratings, delivery times? ### Part 1 — Frame the problem and the serving system Define what the ranker optimizes and outline the path from a query to a ranked list within the latency budget. ```hint Two stages for a reason Think about how many restaurants are eligible for one query and address, and how much model you can afford to run on each. ``` #### What This Part Should Cover - The objective and how it maps to a ranking target - Query understanding, candidate retrieval and ranking stages within a latency budget - Hard filters versus learned scoring ### Part 2 — Features and labels Explain which features you would use and which labels you would train on, and how you would turn logged search sessions into training data. ```hint What the log really says Ask what a click, an order and a skipped result each tell you, and how a result's position changes what they tell you. ``` #### What This Part Should Cover - Feature families: query-restaurant relevance, restaurant quality, logistics, user and context - Label choice and grading, negative examples and position bias - Leakage and point-in-time correctness of features ### Part 3 — Design the online A/B test Design the online experiment for a new ranker: unit of randomization, metrics, duration and analysis. Then address the case where the change is large enough to move a meaningful share of orders between restaurants and areas, so that treatment and control compete for the same couriers. ```hint What the two groups share List what customers in treatment and control share in one market at the same moment, and what that implies for the unit you randomize. ``` #### What This Part Should Cover - Primary metric, guardrails, power and duration for a user-randomized test - Why shared courier supply biases a user-level comparison for large changes - An experiment design that removes the interference, with its analysis and its costs ### What a Strong Answer Covers - A clear objective that balances conversion with delivery quality - A staged retrieval and ranking design within a latency budget - Specific features and graded labels, with position bias and leakage handled - Offline metrics used as a filter, and a planned online test as the decision - Recognition of marketplace interference and an experiment design that fits it ### Follow-up Questions - How would you estimate and correct position bias in the logged clicks? - How would you rank a new restaurant that has no order history? - In a test randomized by region and time window, how would you choose the window length, and what carryover effects worry you? - How would you stop the ranker from steering customers to distant restaurants whose orders then arrive late?