Duplicate-site detection for a web search anti-spam team

You are a product manager on the Google Search team responsible for fighting web spam. How would you detect duplicate websites? Walk through what you would treat as a duplicate and how detection would work across an index that covers the whole web. ```hint Separate matching from judging Finding pages whose content matches is one problem. Deciding which copy deserves to rank, and whether a given copy is abusive at all, is a different one. Work out which signals each step needs. ``` ### Clarifying Questions - Does "duplicate" cover only sites that copy other publishers' content, or also legitimate duplication such as mirrors, syndicated articles and one site served on several hostnames? - Should the verdict be made for individual pages, for whole sites, or both? - Will the result be used to collapse repeated results, to pick which copy to show, or to demote the sites that copy? - Do new pages need to be checked as they are indexed, or is a periodic pass over the index acceptable? ### What a Strong Answer Covers - A working definition of a duplicate site that separates abusive copying from legitimate duplication - A content-comparison method that tolerates small edits and scales to billions of pages without comparing every pair - How the original source is told apart from its copies, and how page-level matches roll up to a site-level judgment - How the detector's output changes what searchers see, and what a false positive costs a legitimate publisher - How precision and recall are measured, and how detection keeps up as spammers change tactics ### Follow-up Questions - A scraper starts rewording copied articles so that exact phrases no longer match. How does detection keep up? - A news publisher reports that its syndicated stories are being treated as duplicates. How would you investigate, and what would you change? - How would you decide whether a new version of the detector improves search quality before launching it?