Explain and Reduce Position Bias in Ranking

Explain position bias in a ranking system that learns from user clicks. How can the position at which an item is displayed distort the apparent relevance signal, and how would you diagnose and reduce that distortion? Discuss a concrete modeling approach, what data it requires, and how you would evaluate whether a revised ranker improves relevance rather than simply reproducing the previous display order. ### What a Strong Answer Covers - The distinction between an item being examined, being relevant, and being clicked. - Why raw click-through rates by position confound presentation effects with the existing ranker's item selection. - A defensible way to estimate position effects and use the estimates during learning, including its assumptions and failure cases. - Evaluation that does not treat every unclicked or unshown item as an irrelevant item. ### Follow-up Questions - What limits arise if the logging system always puts the same items in the same positions? - How would very small examination propensities affect learning or offline evaluation?