Design ML ranking for query suggestions

Q: Design ML ranking for query suggestions

This question evaluates a candidate's competency in designing a production-grade machine learning ranking system for re-ranking autocomplete suggestions, encompassing label definition for long-term success, counterfactual bias correction, feature engineering (including multilingual and Unicode handling), model selection, serving constraints, and evaluation strategies. It is commonly asked in the Machine Learning domain to assess practical application-level understanding of offline/online evaluation, bias mitigation, latency and memory trade-offs, and robustness to feedback loops and distributional drift.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Technical Screen rounds at Etsy.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Etsy during technical interviews.

Question

Re-rank Query Suggestions for Autocomplete

Context

You are building a re-ranking system for search autocomplete. For each keystroke, a candidate generator proposes suggestions; your job is to re-rank them to maximize user success. You have impression-level logs with fields:

user_id, timestamp, locale, device
typed_prefix, suggested_term, position (original rank shown)
clicked (0/1), dwell_time
downstream_query (what the user typed/clicked next)
eventual_success (binary indicator of success later in the session)

Assume suggestions are shown as a slate (top K suggestions) each time a prefix changes.

Tasks

Design the ML system and specify:

Labels for training that best reflect long-term user success (e.g., success within session vs. click), and how to create time-respecting train/validation/test splits to avoid leakage.
How you will correct position/selection bias (e.g., counterfactual logging with propensities, inverse propensity weighting, randomized buckets/interleaving, click models).
Feature sets: contextual, lexical/matching, popularity and time series, embeddings/LM semantics; and how to handle multilingual text and Unicode normalization.
Model class, serving constraints (latency/memory), and fallbacks for cold-start terms/users.
Strategies to limit feedback loops, distribution drift, and unsafe/low-quality suggestions.
Offline and online evaluation plan, including rollback criteria.

Design ML ranking for query suggestions

Quick Overview

Re-rank Query Suggestions for Autocomplete

Context

Tasks

Solution

Comments (0)