Majority-Vote Labels from Three Noisy Annotators, Then Beat the Baseline

You are given a binary classification dataset in which every example was labeled by three annotators, each giving a binary label (0 or 1). The annotators are noisy: their labels sometimes disagree with one another and with the truth. Build the pipeline that turns these noisy labels into training labels, trains on them, and evaluates the result. The original exercise had at least three parts. The report describes two of them, and this practice version keeps those two in order; the part in between was not reported. - Combining each example's three labels into one label. A simple majority vote is enough here; the report advises against over-engineering this step. - The final part (the third in the original exercise): run the pipeline end to end and report its evaluation metrics, which are expected to come out above a baseline the exercise provides. A result below the baseline is the outcome to avoid. The data format, the model, the metrics and the baseline were not reported. Treat these as assumptions to confirm: each row holds an example id, the example's features, and the three annotators' labels; a separate evaluation set has labels you can trust; and you may choose a simple classifier. ### Clarifying Questions - Does every example have exactly three labels, or can a label be missing? - Is there an evaluation set with trusted labels, or are the metrics computed against aggregated labels as well? - Which metrics does the exercise report, such as accuracy, F1 or AUC, and what exactly is the baseline: a fixed score, the same model trained on one annotator's labels, or something else? - Is the model given, or is choosing it part of the task? - Are the three annotators the same people on every example, with their identities in the data, or are the labels anonymous? ### Part 1 — Aggregate the three labels Write the code that turns each example's three binary labels into one label by majority vote, ready to use as a training target. ```hint Three votes, two classes Work out whether a tie is possible with exactly three binary votes and what changes when one label is missing. Also consider whether a unanimous vote and a split vote should be treated the same way later on. ``` #### What This Part Should Cover - A correct majority vote over the three labels, written simply and vectorized where possible - Validation of label values, and handling of missing labels and any ties they create - Keeping the agreement between annotators available for later steps ### Part 2 — Run the pipeline and beat the baseline Train a classifier on the aggregated labels, evaluate it, and report the metrics next to the baseline. If the result does not beat the baseline, find out why and improve it. ```hint Rule out the plumbing first Before changing the model, check that your metric is computed on the same data and in the same way as the baseline's, and that the labels you train on and the labels you evaluate against have not been mixed up. ``` #### Clarifying Questions for this Part - Is there a validation split for tuning, separate from the set the final metrics are reported on? - Is the comparison with the baseline made on one metric, or must every reported metric beat it? #### What This Part Should Cover - A clean split between training and evaluation with no leakage, and metrics computed against trusted labels - Reporting the metrics on the same footing as the baseline - Concrete improvements tied to the label noise when the result falls short, chosen without touching the test set ### What a Strong Answer Covers - Keeps the aggregation simple, as advised, and explains when majority vote reduces label noise and when it cannot - Clean, runnable code that finishes end to end within the interview - Evaluation against trusted labels with a metric suited to the class balance - Uses the agreement between annotators to improve training when the baseline is not beaten - States the final numbers against the baseline clearly, with how they were obtained ### Follow-up Questions - Annotator identities are known, and one annotator is much noisier than the other two. How would you detect that, and what would you change? - Instead of a hard majority label, how could the training use the split of the three votes? - Why can majority vote fail to help when the annotators' mistakes are correlated? - Some examples now have one label and others five. How does the aggregation change?