Human Feedback And Annotation Data Quality

What's being tested

Interviewers probe your ability to measure, diagnose, and mitigate annotation noise so downstream model and experiment conclusions are valid. They want to see statistical rigor (inter-rater reliability, sampling variance), experimental design for labeling (stratified sampling, adjudication), and practical tradeoffs (cost vs. label quality, active relabeling). You are judged on framing clarifying questions, selecting appropriate metrics, running error-budget analyses, and recommending actionable next steps rooted in data.

Core knowledge

Inter-rater reliability: common statistics are Cohen's kappa for two raters and Krippendorff's alpha for many/rate-variable tasks; Cohen's kappa = $(p_o - p_e)/(1-p_e)$ where $p_o$ is observed agreement, $p_e$ expected by chance.
Confusion-matrix diagnostics: compute per-rater confusion matrices vs. a gold or majority label; derive precision/recall/F1 for each annotator to find systematic bias (e.g., conservative positive labeling).
Sampling for quality estimation: use stratified sampling across model score buckets, user cohorts, or time to estimate label error rates with lower variance; compute margin using $SE \approx \sqrt{\hat{p}(1-\hat{p})/n}$ and choose $n$ to meet desired confidence.
Adjudication workflows: common pipelines include majority-vote, expert adjudication (tie-breaker), and probabilistic label aggregation (Dawid–Skene) that estimate per-rater confusion matrices to produce latent true labels.
Label noise models: model noise as class-conditional flip rates or rater-specific confusion matrices; recognizing non-random (systematic) noise matters more than i.i.d. noise for downstream bias.
Impact on metrics: measurement error attenuates effects — estimated treatment effect $\hat{\tau}$ has extra variance and bias; when labels are noisy, power drops and Type I/II error rates shift.
Cost-quality tradeoff: estimate value per reduced error (e.g., expected model metric improvement per additional high-quality label) and optimize budget across sampling, training, and adjudication.
Online vs. offline label drift: track time-series of inter-rater agreement and label distributions; sudden shifts may indicate guideline drift or dataset distributional change requiring prompt retraining or guideline updates.
Active relabeling: prioritize examples near decision boundary, high model uncertainty, or high disagreement among annotators for relabeling to maximize label-information per dollar.
Calibration of evaluation: when creating evaluation sets, enforce label freeze, dedicated annotator pools, and document labeling spec to ensure reproducibility and guard against label leakage.
Hypothesis testing under noisy labels: adjust variance estimates to account for annotation error; use bootstrap or simulation to measure the probability that observed metric deltas could arise from labeling noise.
Metrics to monitor: track label_agreement_rate, per_label_precision/recall, annotator_completion_time, and annotation_spread (entropy across annotators) to diagnose quality and cost tradeoffs.

Worked example — "Design an annotation scheme and evaluate its quality for sentiment labels"

Frame: ask clarifying questions: what is the exact label granularity (binary vs. 5-point), who are annotators (experts vs. crowd), downstream use (training vs. evaluation), and budget/time constraints. Skeleton: (1) define a concise labeling spec with examples and edge cases, (2) pilot-label a stratified sample across domains and model-score buckets, (3) compute inter-rater agreement (Cohen's kappa or Krippendorff's alpha), per-class confusion matrices, and time/cost per label, (4) run an adjudication plan for disagreement cases and estimate post-adjudication label quality and remaining error. Tradeoff: explicitly flag the choice between finer granularity (more informative but lower agreement) and coarser labels (higher agreement but less signal for modeling). Close: propose concrete next steps — increase pilot size if CI wide, run active relabeling on model-uncertain examples, and if time permits, test how model metrics change when trained on adjudicated vs. raw labels.

A second angle — "Diagnose low agreement in human annotations"

Ask if low agreement is uniform across classes, annotators, or data slices. Evaluate per-example disagreement entropy and per-annotator confusion matrices to separate problematic examples (ambiguous content) from problematic annotators (systematic misunderstanding). Consider whether the labeling spec lacks clarity for specific edge cases, whether samples contain adversarial or off-domain content, or whether task cognitive load/time correlates with disagreement. Recommend targeted interventions: refine spec with clear examples, re-train annotators on failure cases, shift to coarse labels for ambiguous slices, or introduce expert adjudication only for contentious subsets.

Common pitfalls

Pitfall: Treating low agreement as purely "bad" and immediately firing annotators — often the root is an ambiguous spec or inherently subjective data; diagnose per-slice disagreement first.

Pitfall: Relying only on overall agreement metrics (e.g., percent agreement) — these mask class imbalance and chance agreement; always report chance-corrected statistics like kappa or alpha.

Pitfall: Ignoring sampling variance when comparing label quality across conditions — failing to compute confidence intervals can lead to overconfident decisions about who is "good" or whether a change improved quality.

Connections

Interviewers may pivot to active learning strategies for label efficiency, fairness and bias analysis when annotations differ across demographic groups, or A/B testing measurement when labels are used to compute product metrics. Be ready to link annotation quality improvements to model evaluation and experiment sensitivity.