Interleaving and Multileaving for Ranking Comparison at Scale

What it is Interleaving blends the results of two ranking models into a single list for a user session and attributes each click to the model that supplied that item. Because users see a shared page, position and user-mix biases largely cancel, making it a fast, low-traffic way to tell which ranker is better. Multileaving generalizes this to compare more than two rankers at once. (microsoft.com)

Why interviewers ask about it At companies with high experiment cost and limited A/B bandwidth (e.g., large e-commerce or social feeds), this technique accelerates iteration on ranking quality. Teams report big speedups versus traditional A/B for detecting small deltas, letting them gate expensive launches or decide which candidates deserve full-rollout tests. (medium.com)

Core ideas to know

A common pitfall Candidates often describe the merge but ignore attribution edge cases. If both rankers propose the same item, failing to assign it to exactly one source causes “credit leakage” and bias. Another miss: not considering collisions in multileaving when the number of rankers approaches the results-per-page, which weakens pairwise discriminability. Finally, people claim it “fixes” all click biases; it only cancels shared-page biases and still needs sensible click models and guardrails. (microsoft.com)

Further reading