# Design human review to estimate model accuracy You need to estimate the **accuracy** of an ML classifier on a population of subjects. You can only afford **K total human reviews**. Each human review produces a **binary judgment** (0/1) for a subject (assume it is intended to represent the “true label,” but reviewers may be noisy). You must choose how to allocate reviews: - **Option 1:** review **K different subjects once each** (1 review per subject). - **Option 2:** review **fewer subjects**, but assign **multiple independent reviews per subject**, and use **majority vote** (or another aggregation). **Question:** Which option is better for estimating the model’s accuracy, and under what assumptions? Provide a statistical argument, discuss bias/variance trade-offs, and propose a practical review design (including how you would quantify uncertainty with a confidence interval). ### Clarifying Questions to Ask - Clarify the random variables, distributional assumptions, independence assumptions, and desired output. - Show enough derivation for the interviewer to follow the reasoning. - Explain how you would validate the result with simulation or sensitivity checks. ### What a Strong Answer Covers - A correct setup with definitions, formulas, and boundary conditions. - A step-by-step derivation or estimation plan. - Interpretation of the result, including uncertainty and practical limitations. - Checks for assumptions, edge cases, and numerical stability. ### Follow-up Questions - How would the result change if the assumptions were relaxed? - Can you verify the answer with a simulation? - What is the most likely source of estimation error?