Design ML ranking for query suggestions
Overview
This question evaluates a candidate's competency in designing a production-grade machine learning ranking system for re-ranking autocomplete suggestions, encompassing label definition for long-term success, counterfactual bias correction, feature engineering (including multilingual and Unicode handling), model selection, serving constraints, and evaluation strategies. It is commonly asked in the Machine Learning domain to assess practical application-level understanding of offline/online evaluation, bias mitigation, latency and memory trade-offs, and robustness to feedback loops and distributional drift.
Re-rank Query Suggestions for Autocomplete
Context
You are building a re-ranking system for search autocomplete. For each keystroke, a candidate generator proposes suggestions; your job is to re-rank them to maximize user success. You have impression-level logs with fields:
- user_id, timestamp, locale, device
- typed_prefix, suggested_term, position (original rank shown)
- clicked (0/1), dwell_time
- downstream_query (what the user typed/clicked next)
- eventual_success (binary indicator of success later in the session)
Assume suggestions are shown as a slate (top K suggestions) each time a prefix changes.
Tasks
Design the ML system and specify:
- Labels for training that best reflect long-term user success (e.g., success within session vs. click), and how to create time-respecting train/validation/test splits to avoid leakage.
- How you will correct position/selection bias (e.g., counterfactual logging with propensities, inverse propensity weighting, randomized buckets/interleaving, click models).
- Feature sets: contextual, lexical/matching, popularity and time series, embeddings/LM semantics; and how to handle multilingual text and Unicode normalization.
- Model class, serving constraints (latency/memory), and fallbacks for cold-start terms/users.
- Strategies to limit feedback loops, distribution drift, and unsafe/low-quality suggestions.
- Offline and online evaluation plan, including rollback criteria.
Solution
Login required