Design large-scale near-duplicate video detection
Company: Google
Role: Machine Learning Engineer
Category: System Design
Difficulty: hard
Interview Round: Onsite
Design a **product-grade fuzzy (near-)duplicate detection** system for a large short-video platform.
You need to detect whether an uploaded video is a near-duplicate (or highly similar) to existing content at very large scale.
Address:
- **Feature/embedding generation** (e.g., vision/audio/text signals).
- Storage and retrieval for similarity search (vector database / ANN index).
- How to handle **very large corpora** (tens/hundreds of millions+ videos) and **product-level QPS**.
- How to support **very high concurrency** while keeping low latency.
- Techniques such as **compression / quantization** and **LSH/ANN**.
- How to reduce and handle **false positives/false negatives**.
- If creators can **appeal** a takedown/duplicate decision: design a **human-in-the-loop** workflow to calibrate and improve the model.
- How to **model and quantify the trade-off** between embedding dimension and end-to-end system latency (and also accuracy/recall).
Quick Answer: This question evaluates a Machine Learning Engineer's competency in scalable similarity search and representation learning for multimedia, including embedding design, retrieval infrastructure, compression/quantization, large-scale ANN indexing, and human-in-the-loop decision workflows.