Improve classifier with noisy multi-annotator labels
Quick Overview
This question evaluates a candidate's ability to analyze noisy multi-annotator labels, design label-aggregation and dataset-splitting strategies, and choose modeling and evaluation approaches for binary text classification.
Improve classifier with noisy multi-annotator labels
Company: OpenAI
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: hard
Interview Round: Technical Screen
## Problem
You are given a **text dataset** for a **binary classification** task (label in \{0,1\}). Each example has been labeled by **multiple human annotators**, and annotators often disagree (i.e., the same item can have conflicting labels).
You need to:
1. Perform a **dataset/label analysis** to understand the disagreement and likely label noise.
2. Propose a **training and evaluation approach** that improves **offline metrics** (e.g., F1 / AUC / accuracy), given the noisy multi-annotator labels.
### Assumptions you may make (state them clearly)
- You have access to: raw text, per-annotator labels, annotator IDs, and timestamps.
- You can retrain models and change the labeling aggregation strategy, but you may have limited or no ability to collect new labels.
### Deliverables
- What analyses would you run and what would you look for?
- How would you construct train/validation/test splits to avoid misleading offline metrics?
- How would you convert multi-annotator labels into training targets?
- What model/loss/thresholding/calibration choices would you try, and why?
- What failure modes and edge cases could cause offline metric gains to be illusory?
Overview: This question evaluates a candidate's ability to analyze noisy multi-annotator labels, design label-aggregation and dataset-splitting strategies, and choose modeling and evaluation approaches for binary text classification.
Community answers
Answer by EL
This is a complete, executable Python simulation. It generates synthetic data, implements the Dawid-Skene algorithm (Expectation-Maximization) to recover signal from noise, trains a model using Soft Targets, and evaluates calibration.
Analysis of the Solution
**Analysis (The
analyze_annotators
function):**
This compares individual annotator agreement against the model's consensus. In the output, you will see annotators with IDs 5, 6, 7 (Spammers) having ~0.50 accuracy, and 8, 9 (Biased) having distinct patterns. This allows us to filter them out in production pipelines.
Splitting Strategy:
I split by
item_id
(
train_ids, test_ids
). This is critical. If we split simply by row, the same text could appear in train (labeled by User A) and test (labeled by User B), causing data leakage.
Label Aggregation (Dawid-Skene):
I implemented the full Expectation-Maximization algorithm.
Why? Majority Vote fails when spammers outnumber experts on a specific item. DS learns a confusion matrix for each worker. It down-weights the spammers (IDs 5-7) automatically because their "error matrix" looks random.
Model & Loss:
I used
BCEWithLogitsLoss
. In PyTorch, this function allows
targets
to be probabilities (floats like 0.8) rather than just integers.
Why Soft Targets? If an item has conflicting labels, it is likely ambiguous. Forcing a hard "1" label tells the model "be 100% sure this is true." Using the DS probability (e.g., 0.65) tells the model "features X suggest a 65% chance of being true,
Answer by Xiaoming
1️⃣ Dataset / Label Analysis — What I Would Actually Run
A. Annotator-Level Analysis (Who Is Noisy?)
You have annotator IDs. That’s gold. Use it.
Per-annotator statistics
For each annotator:
# items labeled
Positive rate
Entropy of their label distribution
Agreement rate with majority
Cohen’s κ against:
Majority vote
Other annotators
Look for:
Systematic bias (always labeling 1)
Low-variance labelers (always 0)
Low agreement but high confidence
Drift over time (timestamps)
If one annotator has:
30% positive rate when global rate is 8%
κ ≈ 0
That’s not disagreement — that’s label corruption.
B. Item-Level Disagreement
For each item:
# annotators
% positive
Label entropy
Variance
Annotator consistency patterns
Define:
p_i = (# positive votes) / (# total votes)
entropy_i = -p_i log p_i - (1-p_i) log (1-p_i)
Plot:
Distribution of p_i
Distribution of entropy
% items with 100% agreement
% items with 50/50 split
If 40% of dataset has p_i in [0.4, 0.6], your dataset is ambiguous — not noisy.
That distinction matters.
C. Time Drift
Using timestamps:
Label distribution over time
Annotator bias over time
Text-topic distribution over time
If annotator A labeled early data and annotator B labeled later data — your “annotator effect” might just be time drift.
D. Text Difficulty vs Disagreement
Train a simple baseline model:
Logistic regression on TF-IDF
Predict majority label
Now:
Identify high-loss examples
Compare to high-entropy examples
If high-loss ≈ high-disagreement → ambiguity.
If
You are given a text dataset for a binary classification task (label in \{0,1\\}). Each example has been labeled by multiple human annotators, and annotators often disagree — the same item can receive conflicting labels.
Your job has two halves:
Perform a
dataset / label analysis
to understand the disagreement and the likely sources of label noise.
Propose a
training and evaluation approach
that improves
offline metrics
(e.g., F1 / AUC / accuracy), given the noisy multi-annotator labels.
This is an open-ended applied-ML design discussion: there is no single "correct" pipeline. The interviewer is looking for how you reason about treating labels as a noisy, structured signal rather than as ground truth, and how you keep your offline evaluation honest.
Constraints & Assumptions
State these explicitly (and any others you add) as you go:
Available signal:
raw text, per-annotator labels, annotator IDs, and label timestamps.
Levers you control:
you can retrain models and change the label-aggregation strategy.
Hard limitation:
you have
limited or no ability to collect new labels
, so you must extract maximum value from the existing annotation redundancy.
The class distribution may be
imbalanced
, and the number of annotators per item may
vary
(some items have one label, others many).
Clarifying Questions to Ask Guidance
A strong candidate scopes the whole problem before designing. Reasonable questions for the interviewer:
How much
redundancy
is there — what's the distribution of annotators-per-item, and how many items have only a single label?
What is the
class balance
, and which error type (false positive vs. false negative) is more costly downstream?
Is the disagreement believed to stem more from
genuinely ambiguous items
or from
a few unreliable annotators
— or is that exactly what we're trying to find out?
Will the
production
input distribution carry annotator IDs, or do we need to generalize to brand-new annotators / unseen text?
Is there any
adjudicated / gold
subset we can trust as ground truth?
Are there known
guideline changes
over the labeling period that could explain temporal drift?
Part 1 — Dataset & label-noise analysis
What analyses would you run, and what would you look for? Specifically, how would you (a) quantify how much annotators disagree, (b) characterize individual annotators, and (c) decide whether a given disagreement reflects real ambiguity vs. a bad labeler?
What This Part Should Cover Guidance
Chance-corrected agreement
rather than raw % agreement, and an awareness of why imbalance inflates the naive number.
Per-item uncertainty
(an empirical positive rate / entropy) used to
rank
items by ambiguity.
Per-annotator reliability
derived from IDs (agreement-vs-consensus, bias toward one class, labeling speed/volume from timestamps).
A concrete test
for separating intrinsic ambiguity from annotator noise (e.g., qualitatively reading high-entropy items, checking whether disagreement concentrates on a few raters).
Part 2 — Splits that don't lie
How would you construct train / validation / test splits so that your offline metrics are not misleading? What is your "ground truth" for the test set when humans themselves disagree?
What This Part Should Cover Guidance
A defensible test ground truth
— restricting the headline test set to high-redundancy, high-agreement (or adjudicated) items.
Leakage control
— grouping by item / near-duplicate text, and isolating annotator-ID or batch artifacts.
Two explicit generalization regimes
— known-annotator vs. new-annotator (group-by-annotator), tied to whether production carries annotator IDs.
Temporal discipline
— splitting forward in time, and fitting any aggregation on the train fold only so test labels don't bleed into training targets.
Part 3 — Turning multi-annotator labels into training targets
How would you convert the multiple per-annotator labels into the targets you actually train on? Lay out the options and give a justified default.
Soft labels as the reframing
for ambiguous items, motivated by what the "right" target is for a 50/50 item.
A justified default
rather than a list, with the trade-offs (redundancy required, complexity, robustness) made explicit.
Part 4 — Model, loss, calibration, and thresholding
What model family, loss, calibration, and thresholding choices would you try, and why?
What This Part Should Cover Guidance
Model choice with a baseline
— a fine-tuned text encoder plus a cheap linear/GBM floor, with the acknowledgment that the label strategy matters more than the architecture.
Loss matched to the target and the imbalance
— soft-target CE for soft labels, class-weighting/focal under skew, noise-robust losses if hard labels are forced.
Noise-aware regularization
— early stopping on a clean fold, recognizing that flexible models memorize noisy labels late.
Explicit calibration and objective-driven thresholding
— temperature/Platt/isotonic + ECE, threshold tuned on a cost or F1 objective, and threshold-free metrics reported alongside.
Part 5 — Why offline gains can be illusory
What failure modes and edge cases could make an offline metric improvement misleading or non-reproducible — and what's your guard for each? Aim for breadth: think across the labels, the splits, the metric itself, the data you may have cleaned, and the gap between offline data and production.
What This Part Should Cover Guidance
Breadth across the named axes
— at least one concrete failure mode each for the labels, the splits, the metric, over-cleaning, and the offline-to-production gap.
A checkable guard per failure mode
, not just naming the risk.
Statistical honesty
— bootstrap/CIs on the test metric, requiring a gain to clear the noise and to hold on more than one metric and on subgroups.
What a Strong Answer Covers Guidance
These dimensions span all parts — an interviewer listens for them throughout, not in any single part:
Separating sources of disagreement
— distinguishing intrinsic item ambiguity from annotator noise/bias, and treating them differently in
every
stage (analysis, targets, evaluation), rather than collapsing labels to ground truth.
Internal consistency across the pipeline
— the target chosen in Part 3 propagates coherently into the loss (Part 4) and the way ambiguous items are scored (Parts 2 and 5), instead of five disconnected answers.
Evaluation-first discipline
— treating the test harness as the load-bearing decision and being skeptical that any offline gain is real.
Follow-up Questions Guidance
Be ready to go deeper:
Your annotator-reliability model gives strange estimates (or won't converge) when redundancy is low or one class dominates. Why does it
degenerate
, and what do you fall back to?
The offline F1 improves by a point, but
online metrics don't move
after deployment. What are the likely explanations, and how would you have caught it earlier?
The production system
won't
carry annotator IDs at inference time. How does that constraint shape your
split design
and your choice of which generalization question to test?