Improve Training With Noisy Annotators
Company: OpenAI
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: hard
Interview Round: Onsite
You are given a labeled training dataset as a Pandas DataFrame. Each row contains features, an observed label, and an annotator identifier. The annotators have varying quality, so some labels may be noisy. You are also given baseline model-training code that trains on the raw dataset and reports validation performance.
Design and implement a data-cleaning or reweighting approach that improves the model's validation performance. Explain how you would:
1. Establish and interpret the baseline.
2. Measure label quality and annotator reliability.
3. Clean, relabel, remove, or reweight examples.
4. Retrain the model and evaluate whether performance improved.
5. Explain basic classification metrics such as precision, recall, and F1 score.
Quick Answer: This question evaluates competency in handling noisy labeled data, estimating annotator reliability, designing cleaning or reweighting strategies, retraining models, and explaining classification metrics such as precision, recall, and F1.