Design logo infringement detection system

### Scenario You work for a large e-commerce company. Brands register their official logos with you (e.g., Nike swoosh, Apple logo, etc.). Third-party sellers upload product images, and some may illegally use these brand logos (counterfeit goods, unauthorized resellers, etc.). You are asked to design an **ML-powered logo infringement detection system**. The interviewer specifically wants a **search-based solution**: given reference logo images, the system should search over product images to find those that contain the logo or are visually very similar. ### Requirements **Functional** 1. Given a new product image, automatically flag whether it likely contains any **protected brand logo**. 2. Allow brand-protection teams to: - Query: "Find all product images similar to this logo image." (image-to-image search) - See ranked search results (most visually similar first). 3. Support **multiple brands** and potentially many logo variants per brand (different colors, orientations, backgrounds). **Non-functional** 1. Scale to **tens of millions** of product images. 2. Low latency for online checks at upload time (e.g., < 500 ms per image for initial screening). 3. High **recall** (don’t miss infringements) while keeping false positives low enough that human reviewers can manage the queue. ### Constraints and assumptions - Each product can have multiple images. - Logos may be **small, rotated, partially occluded**, or overlaid on complex backgrounds. - Adversarial sellers may modify logos slightly (color shifts, aspect ratio changes, adding noise, mirroring, etc.). ### Tasks 1. **High-level architecture** Describe the overall system architecture, including: - Offline indexing pipeline for product images. - Online inference pipeline when a new image is uploaded. - How search is performed given a query logo image. 2. **Representation & model choice** Explain how you will represent images and logos so that visually similar logos are close in some embedding space. Consider: - Backbone architecture (e.g., CNN, Vision Transformer). - Whether you do global image embeddings, local (patch/region) embeddings, or both. - How you handle small logos within large images. 3. **Search system** Given the embeddings, design a search subsystem that can: - Index millions of product image embeddings. - Quickly retrieve the top-K most similar images to a query logo embedding. - Support incremental updates as new products are added. 4. **Training strategy** Describe how you would **train** the model(s): - What labeled data do you need (e.g., logo bounding boxes, positive/negative pairs)? - How would you leverage metric learning (e.g., contrastive or triplet loss) for logo retrieval? - How can you generate additional training data (e.g., synthetic logo overlays)? 5. **Decision logic & thresholds** Once you retrieve candidate matches, how do you decide whether an image truly contains an infringing logo? - How do you combine similarity scores, brand-specific thresholds, and possibly a second-stage classifier or detector? - How would you handle different brands having different risk tolerances (e.g., very strict brands vs. more lenient ones)? 6. **Evaluation & monitoring** Propose metrics and an evaluation strategy: - How do you measure performance (precision, recall, ROC/PR curves) both for retrieval and final decisions? - How do you monitor the system in production and collect feedback from human reviewers to improve the models? 7. **Scaling & extensions** Discuss how you would: - Scale the system as the catalog grows (index sharding, approximate nearest neighbor search, caching). - Handle updates when new brands or new logo variants are registered. - Deal with adversarial attacks and continuously evolving counterfeit techniques. Your answer should walk through the end-to-end pipeline and clearly separate **ML components** (modeling, training, embeddings) from **system components** (storage, search index, services, monitoring).