Design a Search Ranker: Feature Inputs, Model Design, Training, Hosting and Evaluation

Design a search ranker: given a user's query and a set of candidate results, the ranker decides the order in which the results are shown. This was the small system design section of a technical screen for a Staff machine learning engineer, and a complete answer covers five areas: the feature inputs, the design of the ranker, training, hosting, and evaluation. ```hint Decide what "good" means first Before choosing a model, decide which logged user behavior counts as a good result for a query, and how a result's position on the page distorts that signal. ``` ```hint Spend the latency budget wisely Think about how many candidates you can afford to score with your most expensive features and model within the latency budget, and what happens to the rest. ``` ### Constraints and Clarifications - Not given: what is being searched, the query volume, the corpus size, the latency budget or the business objective. Ask, or state your assumptions. ### Clarifying Questions - What is being searched (for example people, jobs, posts or documents), and should the ranking be personalized to the searcher? - What does success mean: clicks, longer engagement, a downstream action such as an application or a message, or a blend? Which metrics must not regress? - Is candidate retrieval in scope, or does the ranker receive a candidate list from an existing retrieval system? - What query volume, corpus size and latency budget should the design meet? - How quickly must new items, and the searcher's latest activity, affect the ranking? ### What a Strong Answer Covers - Scopes the product, the objective and the latency budget, and turns the objective into concrete training labels - Feature inputs: query, item, searcher, query-item match and context features, with online and offline values kept consistent - Ranker design: stages sized to the latency budget, the model at each stage, and the choice between pointwise, pairwise and listwise objectives - Training data built from logs, with position bias, sampling and time-based leakage handled, and a retraining cadence - Hosting: feature fetching, scoring within the latency budget, caching, fallbacks and a safe rollout path - Evaluation: offline ranking metrics tied to online A/B metrics and guardrails, plus monitoring after launch - Staff-level judgment about what to build first and where the biggest risks lie ### Follow-up Questions - Offline NDCG improves but the A/B test is flat. How do you diagnose the gap? - Newly created items have no engagement history. How does the ranker treat them? - The latency budget is cut in half. What do you change first? - How do you stop the ranker from learning mostly from its own past rankings?