Search Ranking Evaluation And Interleaving

What's being tested

This topic tests whether you can evaluate search ranking changes with metrics that reflect user utility rather than merely counting clicks. You should be able to distinguish offline relevance evaluation from online experimentation, explain why standard A/B tests can require substantial traffic, and use interleaving to compare rankers efficiently within the same search session. Google cares because small ranking changes affect billions of impressions, and a Data Scientist must separate genuine relevance improvements from position bias, novelty effects, query mix changes, and noisy user behavior.

Core knowledge

• Ranking evaluation starts by defining the unit of analysis: query, query-user session, impression, or task completion. Offline metrics such as NDCG@k, MRR, MAP, and recall measure different objectives; online metrics such as click-through rate, reformulation rate, abandonment, and long-click rate capture realized behavior.

• Discounted cumulative gain values highly relevant documents near the top:
  $DCG@k=\sum_{i=1}^{k}\frac{2^{rel_i}-1}{\log_2(i+1)}$
  Normalize by the ideal ordering to obtain NDCG@k; use graded labels when “excellent” and “acceptable” results should not count equally.

• Interleaving combines results from two rankers into one displayed list, then attributes user interactions to the ranker contributing the clicked result. It controls much of the query and user heterogeneity because both rankers compete within the same impression rather than across separate users.

• Team Draft interleaving alternates turns: each ranker selects its highest-ranked document not already included, with a randomized first picker. It is simple and interpretable, but duplicate results, unequal candidate overlap, and position-dependent click behavior can create attribution bias.

• Balanced Interleaving or probabilistic interleaving samples among valid combined lists so each ranker receives comparable exposure. The design should define how probabilities are computed and use inverse-probability weighting if displayed positions or assignments are not equally likely.

• The primary interleaving outcome is often a win rate or preference statistic:
  $\hat{p}=\frac{\text{wins}_A+\frac{1}{2}\text{ties}}{\text{wins}_A+\text{wins}_B+\text{ties}}$
  Analyze at the query or session level, not as independent clicks, because one session may generate multiple correlated interactions.

• Position bias means users are more likely to click higher-ranked results even when relevance is identical. Interleaving reduces common exposure differences, but it does not eliminate bias; inspect clicks by position, use randomized first selection, and consider click models such as examination models or propensity weighting.

• A click is an imperfect relevance label: attractive titles, navigational intent, brand familiarity, and result snippets affect it. Pair click outcomes with long clicks, rapid backtracks, query reformulations, dwell time, task completion, explicit judgments, or satisfaction surveys, while recognizing that dwell time itself can be confounded by page type.

• Statistical inference must match the randomization unit. Use paired analyses or clustered standard errors by query, user, or session; report confidence intervals and practical effect sizes. A tiny statistically significant preference may not justify latency, infrastructure, or product risk.

• Sample-ratio mismatch and instrumentation checks precede interpretation. Verify traffic assignment, ranker eligibility, logging completeness, duplicate handling, bot filtering, and treatment exposure before trusting a preference estimate. A missing click can look like a ranker loss.

• Query heterogeneity matters: navigational, informational, transactional, head, tail, fresh, multilingual, and mobile queries can respond differently. Pre-specify segments, control multiple comparisons, and report both aggregate and segment-level effects so a gain on head queries does not conceal tail-query harm.

• Offline-online disagreement is expected. Human labels may reward topical relevance while users reward freshness or authority; static test sets miss query drift and feedback loops. Use offline metrics for fast iteration, interleaving for sensitive pairwise comparison, and randomized A/B tests for final product-level impact.

Worked example — “Compare Two Search Rankers Using Interleaving”

A strong candidate would first clarify whether the goal is pairwise relevance comparison, launch approval, or diagnosis, and whether the unit of randomization is query, session, or user. They would state assumptions: both rankers return eligible results for the same query, clicks are logged reliably, and the primary outcome is a pre-specified session-level preference rather than raw click count. The answer would be organized around four pillars: candidate-list construction, an interleaving algorithm, unbiased outcome attribution, and statistical analysis with guardrails. They might choose Team Draft for interpretability, randomize the first picker, and define deterministic duplicate handling before looking at results. They would explicitly flag the tradeoff between simple attribution and exposure bias, noting that probabilistic interleaving may reduce bias but be harder to explain and implement. They would compare win rate, tie rate, clicks per session, reformulations, and latency while checking query-type and device segments. They would validate logging and run an A/A test to detect assignment or attribution problems. If they had more time, they would calibrate the result against judged NDCG@10, investigate disagreement queries, and follow with a properly powered user-level A/B test.

A second angle — “Diagnose a Ranking Metric Mismatch”

Here the same concept is framed around a case where NDCG@10 improves but interleaving preference and long-click rate decline. The candidate should ask whether the gains are concentrated in a narrow query segment, whether labels are stale, and whether the new ranker changes result diversity or freshness. They should inspect position-specific clicks, reformulations, abandonment, and query-level paired outcomes rather than declaring the online metric wrong. A plausible explanation is that the offline labels reward topical relevance while users prefer direct answers or authoritative destinations for current queries. The next step is to identify disagreement cohorts, obtain targeted human judgments, and run a controlled experiment with guardrail metrics.

Common pitfalls

Pitfall: Saying “the ranker with more clicks wins” ignores exposure, duplicate results, position bias, and session correlation. Define attribution before observing outcomes, use the randomization unit for inference, and report uncertainty around a paired preference estimate.

Pitfall: Presenting one aggregate win rate as a launch decision hides important heterogeneity. A stronger answer names the primary metric, guardrails, minimum practical effect, key cohorts, and the conditions under which the decision would change.

Pitfall: Treating interleaving as a replacement for an A/B test overstates what it identifies. Interleaving is excellent for sensitive pairwise ranker comparison, but it does not fully estimate absolute product impact, latency effects, ecosystem effects, or long-term behavior; validate promising results with randomized experimentation.

Connections

An interviewer may pivot to counterfactual evaluation, click models, A/B test power analysis, multiple-hypothesis correction, or human relevance judgments. They may also ask how ranking quality interacts with latency, diversity, freshness, fairness, and personalization.

Further reading

• “Unbiased Learning-to-Rank with Biased Feedback” — Joachims et al. — foundational treatment of position bias and propensity-based correction.

• “Efficient Online Evaluation of Ranking Systems” — Chapelle et al. — practical analysis of interleaving and online ranker comparison.

• “Search Result Diversification” — Santos, Macdonald, and Ounis — useful context for evaluating relevance when multiple intents and result types matter.