Design a machine learning system that **mines novel or interesting images** from a massive, unlabeled image corpus. The corpus is far too large for exhaustive human review, so the system must surface a small, high-value stream of candidates automatically. You may use human labelers, but the system **must not depend heavily on manual labeling** — labels are a scarce, expensive resource to be spent strategically. The output is a continuously updated, ranked, de-duplicated set of "interesting" images that researchers or downstream training pipelines can consume. Cover the full system end to end: 1. How to **define and measure** novelty / interestingness when there is no label. 2. **Data ingestion, deduplication, filtering, and storage** at large scale. 3. **Model choices** for image representation and scoring. 4. How to **use a limited human-label budget** effectively. 5. How to **rank, diversify, and sample** candidate images. 6. **Evaluation metrics and online monitoring** — how do you know it works? 7. **Scaling considerations** for a very large, growing corpus. ```hint Where to start — frame the target "Interesting" is not one thing. Before scoring anything, ask what would make an image valuable to *this* consumer, and whether a single number can capture it. A pure outlier score has a known failure: the rarest images in a web corpus are often the junk (corrupt, blurry, watermarked, screenshots, unsafe), so think about what you'd need to *separate out* from genuine novelty. ``` ```hint Representation & rarity without labels With no labels, you still have geometry. If a frozen encoder mapped every image to a vector, what could the *arrangement* of those vectors tell you about which images are rare versus common (think nearest-neighbor distance, local density, cluster size)? Also weigh what you gain from an encoder trained to align images with **text** versus one trained on images alone, when you later want to steer or audit what you've surfaced. ``` ```hint Spending the human budget You can't label millions, so which handful of images, if labeled this round, would teach a model the most? Think about where a model is least sure versus where it's already confident, and about covering regions you haven't sampled yet (active learning). Separately: for a subjective target, is asking "rate this 1–5" the most reliable thing you can ask a human, or is there a question format that's easier to answer consistently? ``` ```hint Don't return 100 near-identical outliers Ranking purely by an individual score tends to collapse the output onto one rare cluster. The output is a *set*, so its value depends on what the items look like *together*, not just each item's score. What would you add to the selection step — e.g. a redundancy penalty or per-cluster quotas — so the final batch spreads across the rare regions instead of piling into the single rarest one? ``` ### Constraints & Assumptions - **Corpus scale:** assume on the order of $10^8$–$10^9$ images, continuously growing via streaming ingestion. - **Unlabeled:** essentially no ground-truth labels exist up front; "interestingness" is not given and must be defined. - **Human budget:** a small labeling team (think hundreds to low-thousands of judgments per day), not millions — labels are the bottleneck. - **No single ground truth for "interesting":** the target is subjective and product-defined; the spec is yours to propose and defend. - **Quality/safety matter:** raw outliers include corrupted files, screenshots, watermarks, and unsafe content — these are *not* the "interesting" images we want, and unsafe content must be filtered before any human sees it. - **Latency:** this is a batch / near-real-time mining system (minutes-to-hours freshness), not a low-latency online serving path; throughput and label efficiency matter more than per-image latency. - State any further assumptions explicitly (available foundation models, fine-tune vs. inference-only). ### Clarifying Questions to Ask - Who consumes the output, and for what — model-training data acquisition, content discovery, or trust-and-safety auditing? (This redefines "interesting.") - Is this a one-time mining pass over a fixed corpus, or a continuously running pipeline over streaming new data? - What foundation models / embedding encoders are we allowed to use, and can we fine-tune them or only run inference? - Is novelty defined **relative to a reference distribution** (our existing training set) or absolute, and do we have that reference corpus? - What is the daily human-labeling capacity, what label types are allowed (binary / multi-class / pairwise), and what expertise do labelers have? - Are there hard policy/safety constraints on what may be surfaced or stored, and is there a mandated safety classifier? ### What a Strong Answer Covers - A **product-grounded definition** of interestingness broken into measurable sub-scores (rarity, usefulness, quality, safety, diversity), not a single hand-wavy metric. - A clear **candidate-generation vs. re-ranking** separation (cheap recall over billions → expensive scoring on a shortlist). - Concrete **representation choices** with justification (why a VLM embedding, what a perceptual hash is for, what quality/safety features add). - An **active-learning loop** that explicitly economizes labels and improves iteratively. - **Diversity / anti-redundancy** handling at the set level, not just top-K by score. - **Evaluation** that ties to both human judgment *and* downstream value, plus production health metrics. - **Large-scale systems** thinking: ANN indexing, distributed embedding compute, streaming + batch, reproducibility (versioned embeddings). - **Failure modes and mitigations** (bias, near-dup flooding, reviewer overfitting, drift). ### Follow-up Questions - Your novelty score keeps surfacing screenshots and watermarked stock photos. Walk through exactly where in the pipeline you'd catch this and how. - The downstream team says mined images *look* novel but don't improve model accuracy. How would you redesign the objective to optimize for downstream value directly? - After three months the embedding distribution has drifted and the rare clusters from month one now look common. How does the system adapt without re-labeling everything? - How would you A/B test or otherwise causally measure that this mining system beats random / recent-sampling baselines?