Choose 200 of a Million Records to Label, Improving on Uniform Random Sampling

A dataset holds about one million unlabeled records, and only 200 of them can be selected for labeling. The request is to pick the "useful" data, but usefulness is never defined. The task was given verbally, and several facts came out only under questioning: - A uniform random sample of 200 records was accepted as a valid answer but described as not optimal. - Proposals based on access recency or frequency (in the style of least-recently-used or most-frequently-used rules), or on picking records near the mean, median, 75th percentile, or 99th percentile, drew only the reply that the selection is called once. - No definition of usefulness, sampling method, or feature set was provided when asked. Propose a selection method that improves on uniform random sampling. Say what "better than random" means under your stated assumptions, and explain how your method works at this scale when the selection is called only once. ```hint Where a random sample wastes labels Think about what a 200-record uniform sample does to small subgroups and to near-duplicate records. ``` ```hint What a single call rules out If all 200 records are chosen in one call, you cannot label a few, learn from them, and then choose the rest. ``` ### Constraints and Clarifications - No feature representation, model, labeling objective, or evaluation metric is supplied. Any representation you build from the records' fields is your own stated assumption. - The selection is called once over a fixed dataset of about one million records and returns exactly 200 distinct records. ### Clarifying Questions - Will the labels train a model, estimate how often something occurs in the full dataset, or both? - Which fields or metadata can be read before labeling, and are there known categories or sources? - Is there any usage or access history for the records, or does "called once" mean each record is seen only in this one selection? - Can the data be read more than once, or must selection happen in one streaming pass with bounded memory? - Are duplicate or near-duplicate records present? ### What a Strong Answer Covers - A concrete reading of "useful" and of what it means to beat the random baseline. - A specific single-call selection method, the representation it depends on, and its cost at one million records. - Selection bias, reweighting when the labels are also used for estimation, reproducibility, and how to show the method beats random. ### Follow-up Questions - How would your approach change if a trained model existed and the selection could be repeated in rounds? - If the 200 labels must also estimate a rate across all one million records, how do you correct for non-uniform selection? - How would you select the 200 records if the data arrives as a stream that can be read only once?