Compute precision under noisy annotators
Company: Google
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Technical Screen
Two independent annotators review videos. For truly illegal videos, each annotator labels “illegal” with sensitivity s = 0.80; for legal videos, each says “illegal” with false‑positive rate f = 0.02 (specificity 0.98). The base rate of illegal videos is p = 0.01.
- If you flag a video as illegal when at least one annotator says “illegal”, compute the precision (PPV) and recall (TPR).
- If you instead require both annotators to say “illegal”, compute PPV and TPR.
- Which policy yields the higher F1? Show calculations. Generalize formulas for n annotators with majority vote and arbitrary p, s, and f.
Quick Answer: This question evaluates understanding of statistical performance metrics and label-noise propagation by requiring computation of precision, recall, and F1 for binary labels under known annotator sensitivity/specificity and majority-vote aggregation, in the Statistics & Math domain.