Improve classifier with noisy multi-annotator labels

Quick Overview

This question evaluates a candidate's ability to analyze noisy multi-annotator labels, design label-aggregation and dataset-splitting strategies, and choose modeling and evaluation approaches for binary text classification.

Improve classifier with noisy multi-annotator labels

Company: OpenAI

Role: Machine Learning Engineer

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

## Problem You are given a **text dataset** for a **binary classification** task (label in \{0,1\}). Each example has been labeled by **multiple human annotators**, and annotators often disagree (i.e., the same item can have conflicting labels). You need to: 1. Perform a **dataset/label analysis** to understand the disagreement and likely label noise. 2. Propose a **training and evaluation approach** that improves **offline metrics** (e.g., F1 / AUC / accuracy), given the noisy multi-annotator labels. ### Assumptions you may make (state them clearly) - You have access to: raw text, per-annotator labels, annotator IDs, and timestamps. - You can retrain models and change the labeling aggregation strategy, but you may have limited or no ability to collect new labels. ### Deliverables - What analyses would you run and what would you look for? - How would you construct train/validation/test splits to avoid misleading offline metrics? - How would you convert multi-annotator labels into training targets? - What model/loss/thresholding/calibration choices would you try, and why? - What failure modes and edge cases could cause offline metric gains to be illusory?

Overview: This question evaluates a candidate's ability to analyze noisy multi-annotator labels, design label-aggregation and dataset-splitting strategies, and choose modeling and evaluation approaches for binary text classification.

Community answers

Answer by EL

This is a complete, executable Python simulation. It generates synthetic data, implements the Dawid-Skene algorithm (Expectation-Maximization) to recover signal from noise, trains a model using Soft Targets, and evaluates calibration. Analysis of the Solution **Analysis (The analyze_annotators function):** This compares individual annotator agreement against the model's consensus. In the output, you will see annotators with IDs 5, 6, 7 (Spammers) having ~0.50 accuracy, and 8, 9 (Biased) having distinct patterns. This allows us to filter them out in production pipelines. Splitting Strategy: I split by item_id ( train_ids, test_ids ). This is critical. If we split simply by row, the same text could appear in train (labeled by User A) and test (labeled by User B), causing data leakage. Label Aggregation (Dawid-Skene): I implemented the full Expectation-Maximization algorithm. Why? Majority Vote fails when spammers outnumber experts on a specific item. DS learns a confusion matrix for each worker. It down-weights the spammers (IDs 5-7) automatically because their "error matrix" looks random. Model & Loss: I used BCEWithLogitsLoss . In PyTorch, this function allows targets to be probabilities (floats like 0.8) rather than just integers. Why Soft Targets? If an item has conflicting labels, it is likely ambiguous. Forcing a hard "1" label tells the model "be 100% sure this is true." Using the DS probability (e.g., 0.65) tells the model "features X suggest a 65% chance of being true,

Answer by Xiaoming

1️⃣ Dataset / Label Analysis — What I Would Actually Run A. Annotator-Level Analysis (Who Is Noisy?) You have annotator IDs. That’s gold. Use it. Per-annotator statistics For each annotator: # items labeled Positive rate Entropy of their label distribution Agreement rate with majority Cohen’s κ against: Majority vote Other annotators Look for: Systematic bias (always labeling 1) Low-variance labelers (always 0) Low agreement but high confidence Drift over time (timestamps) If one annotator has: 30% positive rate when global rate is 8% κ ≈ 0 That’s not disagreement — that’s label corruption. B. Item-Level Disagreement For each item: # annotators % positive Label entropy Variance Annotator consistency patterns Define: p_i = (# positive votes) / (# total votes) entropy_i = -p_i log p_i - (1-p_i) log (1-p_i) Plot: Distribution of p_i Distribution of entropy % items with 100% agreement % items with 50/50 split If 40% of dataset has p_i in [0.4, 0.6], your dataset is ambiguous — not noisy. That distinction matters. C. Time Drift Using timestamps: Label distribution over time Annotator bias over time Text-topic distribution over time If annotator A labeled early data and annotator B labeled later data — your “annotator effect” might just be time drift. D. Text Difficulty vs Disagreement Train a simple baseline model: Logistic regression on TF-IDF Predict majority label Now: Identify high-loss examples Compare to high-entropy examples If high-loss ≈ high-disagreement → ambiguity. If

Answer by rupalsharmadel

This is a complete
OpenAI logo
OpenAI
Feb 11, 2026
hardMachine Learning EngineerTechnical ScreenMachine Learning
1040
0

Problem

You are given a text dataset for a binary classification task (label in \{0,1\\}). Each example has been labeled by multiple human annotators, and annotators often disagree — the same item can receive conflicting labels.

Your job has two halves:

  1. Perform a dataset / label analysis to understand the disagreement and the likely sources of label noise.
  2. Propose a training and evaluation approach that improves offline metrics (e.g., F1 / AUC / accuracy), given the noisy multi-annotator labels.

This is an open-ended applied-ML design discussion: there is no single "correct" pipeline. The interviewer is looking for how you reason about treating labels as a noisy, structured signal rather than as ground truth, and how you keep your offline evaluation honest.

Constraints & Assumptions

State these explicitly (and any others you add) as you go:

  • Available signal: raw text, per-annotator labels, annotator IDs, and label timestamps.
  • Levers you control: you can retrain models and change the label-aggregation strategy.
  • Hard limitation: you have limited or no ability to collect new labels , so you must extract maximum value from the existing annotation redundancy.
  • The class distribution may be imbalanced , and the number of annotators per item may vary (some items have one label, others many).

Clarifying Questions to Ask Guidance

A strong candidate scopes the whole problem before designing. Reasonable questions for the interviewer:

  • How much redundancy is there — what's the distribution of annotators-per-item, and how many items have only a single label?
  • What is the class balance , and which error type (false positive vs. false negative) is more costly downstream?
  • Is the disagreement believed to stem more from genuinely ambiguous items or from a few unreliable annotators — or is that exactly what we're trying to find out?
  • Will the production input distribution carry annotator IDs, or do we need to generalize to brand-new annotators / unseen text?
  • Is there any adjudicated / gold subset we can trust as ground truth?
  • Are there known guideline changes over the labeling period that could explain temporal drift?

Part 1 — Dataset & label-noise analysis

What analyses would you run, and what would you look for? Specifically, how would you (a) quantify how much annotators disagree, (b) characterize individual annotators, and (c) decide whether a given disagreement reflects real ambiguity vs. a bad labeler?

What This Part Should Cover Guidance

  • Chance-corrected agreement rather than raw % agreement, and an awareness of why imbalance inflates the naive number.
  • Per-item uncertainty (an empirical positive rate / entropy) used to rank items by ambiguity.
  • Per-annotator reliability derived from IDs (agreement-vs-consensus, bias toward one class, labeling speed/volume from timestamps).
  • A concrete test for separating intrinsic ambiguity from annotator noise (e.g., qualitatively reading high-entropy items, checking whether disagreement concentrates on a few raters).

Part 2 — Splits that don't lie

How would you construct train / validation / test splits so that your offline metrics are not misleading? What is your "ground truth" for the test set when humans themselves disagree?

What This Part Should Cover Guidance

  • A defensible test ground truth — restricting the headline test set to high-redundancy, high-agreement (or adjudicated) items.
  • Leakage control — grouping by item / near-duplicate text, and isolating annotator-ID or batch artifacts.
  • Two explicit generalization regimes — known-annotator vs. new-annotator (group-by-annotator), tied to whether production carries annotator IDs.
  • Temporal discipline — splitting forward in time, and fitting any aggregation on the train fold only so test labels don't bleed into training targets.

Part 3 — Turning multi-annotator labels into training targets

How would you convert the multiple per-annotator labels into the targets you actually train on? Lay out the options and give a justified default.

What This Part Should Cover Guidance

  • A spectrum of options — majority vote → soft/probabilistic labels → confidence weighting → annotator-quality–aware aggregation (e.g., Dawid–Skene/EM) → joint end-to-end annotator modeling.
  • Soft labels as the reframing for ambiguous items, motivated by what the "right" target is for a 50/50 item.
  • A justified default rather than a list, with the trade-offs (redundancy required, complexity, robustness) made explicit.

Part 4 — Model, loss, calibration, and thresholding

What model family, loss, calibration, and thresholding choices would you try, and why?

What This Part Should Cover Guidance

  • Model choice with a baseline — a fine-tuned text encoder plus a cheap linear/GBM floor, with the acknowledgment that the label strategy matters more than the architecture.
  • Loss matched to the target and the imbalance — soft-target CE for soft labels, class-weighting/focal under skew, noise-robust losses if hard labels are forced.
  • Noise-aware regularization — early stopping on a clean fold, recognizing that flexible models memorize noisy labels late.
  • Explicit calibration and objective-driven thresholding — temperature/Platt/isotonic + ECE, threshold tuned on a cost or F1 objective, and threshold-free metrics reported alongside.

Part 5 — Why offline gains can be illusory

What failure modes and edge cases could make an offline metric improvement misleading or non-reproducible — and what's your guard for each? Aim for breadth: think across the labels, the splits, the metric itself, the data you may have cleaned, and the gap between offline data and production.

What This Part Should Cover Guidance

  • Breadth across the named axes — at least one concrete failure mode each for the labels, the splits, the metric, over-cleaning, and the offline-to-production gap.
  • A checkable guard per failure mode , not just naming the risk.
  • Statistical honesty — bootstrap/CIs on the test metric, requiring a gain to clear the noise and to hold on more than one metric and on subgroups.

What a Strong Answer Covers Guidance

These dimensions span all parts — an interviewer listens for them throughout, not in any single part:

  • Separating sources of disagreement — distinguishing intrinsic item ambiguity from annotator noise/bias, and treating them differently in every stage (analysis, targets, evaluation), rather than collapsing labels to ground truth.
  • Internal consistency across the pipeline — the target chosen in Part 3 propagates coherently into the loss (Part 4) and the way ambiguous items are scored (Parts 2 and 5), instead of five disconnected answers.
  • Evaluation-first discipline — treating the test harness as the load-bearing decision and being skeptical that any offline gain is real.

Follow-up Questions Guidance

Be ready to go deeper:

  • Your annotator-reliability model gives strange estimates (or won't converge) when redundancy is low or one class dominates. Why does it degenerate , and what do you fall back to?
  • The offline F1 improves by a point, but online metrics don't move after deployment. What are the likely explanations, and how would you have caught it earlier?
  • The production system won't carry annotator IDs at inference time. How does that constraint shape your split design and your choice of which generalization question to test?