Design a Text-to-Image Retrieval System
Company: Gimlet Labs
Role: Member of Technical Staff, ML Systems
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
## Interview Prompt
Design a system where images are uploaded offline and users enter natural-language
questions or descriptions to retrieve the most relevant images. Cover ingestion,
multimodal representations, metadata and access control, vector search, reranking,
evaluation, updates, and failure cases such as duplicate or unsafe images.
### Constraints & Assumptions
- Images can be large and arrive in batches.
- Text queries must retrieve visual concepts even when no matching caption was supplied.
- Results must respect tenant and object-level permissions.
- An image update or deletion must propagate to every derived representation.
### Clarifying Questions to Ask
- Is relevance based on visual content, OCR text, metadata, or all three?
- What corpus size, query rate, and freshness target apply?
- Can the system use human relevance feedback?
### What a Strong Answer Covers
- Safe object ingestion, deduplication, preprocessing, OCR/caption enrichment, and versioned provenance.
- A text-image embedding strategy with metadata filters and optional lexical retrieval.
- Approximate nearest-neighbor candidate search followed by a more accurate reranker.
- Offline recall/precision metrics plus online engagement or task-success signals without conflating them.
- Permission-safe indexing, deletion, moderation, drift monitoring, and fallbacks.
### Follow-up Questions
- How would you handle a query for text visible inside an image?
- How would you diversify results so near-duplicate images do not fill the page?
- What changes if users can submit an image as the query?
Overview: Design permission-aware text-to-image retrieval with safe ingestion, multimodal embeddings, metadata filtering, approximate search, reranking, relevance evaluation, versioned updates, deletion, and duplicate control.