This question evaluates debugging and practical implementation skills for transformer-based text classification, covering model correctness, training loop integrity, data preprocessing, and evaluation metrics within the Machine Learning / Natural Language Processing domain.
##### Question
You are given a transformer-based ML model with four failing unit tests—two bugs are known, two are novel. Identify, debug, and fix each bug so the model trains and evaluates correctly. Given a labeled dataset, write code to train a classifier, analyze the dataset (class balance, feature distributions), and report key performance metrics.
Quick Answer: This question evaluates debugging and practical implementation skills for transformer-based text classification, covering model correctness, training loop integrity, data preprocessing, and evaluation metrics within the Machine Learning / Natural Language Processing domain.
Debug and Fix a Transformer Text Classifier, Then Train and Evaluate It
You inherit a small codebase for a transformer-based text classifier. It ships with four failing unit tests:
Two "known" bugs
— previously documented issues.
Two "novel" bugs
— unexpected, undocumented issues.
Your job is to get the model training and evaluating correctly, and to demonstrate a robust training/evaluation pipeline on a labeled dataset. This is a live coding/discussion exercise: narrate your debugging process out loud, justify each fix, and back every fix with a regression test.
Assumptions (so the task is self-contained)
Language:
Python 3.10+
Libraries:
PyTorch, Hugging Face Transformers, scikit-learn, pandas, numpy
Dataset:
a CSV file with columns
text
(string) and
label
(int) — single-label classification with
K
classes.
Constraints & Assumptions
The encoder is a standard pretrained sentence encoder (e.g., a BERT/DistilBERT-family model); you may pick the checkpoint.
The dataset is small-to-medium and may be
class-imbalanced
— assume it does not fit any "balanced 50/50" expectation.
Single-label, single-text input (no pairs, no multi-label);
K
is not fixed and may exceed 2.
You have CPU and may or may not have a GPU; the code must run correctly on either.
"All tests pass" is the bar for Part A; for Part B, you must produce defensible metrics, not just a number.
Part A — Diagnose and fix the bugs
Identify the root cause of each failing test (2 known, 2 novel) and fix the model/training code so all four tests pass. Then provide a clean, minimal reference implementation of the model and training loop that avoids these bugs.
What a Strong Answer Covers Guidance
A
deterministic triage loop
: reproduce in isolation → classify the failure → minimize → fix one bug → re-run all tests → add a regression assertion.
Whether the
encoder is wired correctly
end-to-end and produces a sentence representation of the
right shape
for the classifier head — i.e., the candidate reasons about what signal the encoder should receive and what the head should consume.
Whether the
loss matches the problem type
the candidate has identified, and whether labels are in a form that loss can consume — judged on the candidate's reasoning, not on a specific criterion you're looking for.
A
split strategy
that keeps evaluation meaningful on small/imbalanced data.
A
regression test per fix
that genuinely distinguishes the bug from the fix (e.g., not a check that passes for both the buggy and correct version).
A reference implementation that is
minimal and reusable
, not a copy of the buggy code with patches bolted on.
Part B — Dataset analysis, training, and evaluation
Given a labeled dataset, analyze class balance and basic feature distributions (e.g., text length, token frequency), then train the classifier. Report key performance metrics — accuracy, precision, recall, F1, plus ROC-AUC when the task is binary — and include guardrails for class imbalance and reproducibility.
What a Strong Answer Covers Guidance
EDA that drives decisions
: class counts/proportions + imbalance ratio, an
untruncated
length distribution and truncation rate, token-frequency / vocabulary signal, and basic data-quality checks (duplicates, conflicting labels).
A
leakage-free split
and class-weight/resampling derived from train data only.
A training loop with the
standard fine-tuning hygiene
(learning-rate schedule, gradient clipping, model selection on the right metric, dynamic padding).
Imbalance-aware metric selection
with the confusion matrix and a clear rationale for which metric leads.
A concrete, named
reproducibility setup
(seeds, deterministic kernels, pinned versions) and the trade-off it implies.
A short menu of
class-imbalance remedies
in a sensible order, validated on the natural (un-resampled) distribution.
Clarifying Questions to Ask Guidance
Where are the four failing tests and the buggy code — do I have the actual test bodies, or just the symptom descriptions? Are the "known" bugs documented anywhere I can read?
How many classes does the dataset have, and is it binary or multiclass? (This decides single-label vs multi-label loss and whether ROC-AUC is a single number or one-vs-rest.)
How imbalanced is the data, and what's the real-world cost of a false positive vs false negative? (This decides the headline metric and whether to weight/resample.)
What's the latency/compute budget and the target sequence length? (This decides
max_length
, the encoder checkpoint, and whether truncation is acceptable.)
Am I optimizing to
pass the tests
(correctness) or to
maximize a held-out metric
(performance), and is there a runtime/GPU constraint I should code against?
Is reproducibility a hard requirement (exact run-to-run repeatability) or is "seeded but allowed to vary slightly" acceptable?
Follow-up Questions Guidance
Suppose a fifth test starts failing only on GPU but passes on CPU — what classes of bugs does that point to, and how would you isolate it?
The dataset grows 100x and many documents exceed
max_length
. What breaks first, and how do you change the model/data pipeline to handle long inputs without silently dropping signal?
Two of your "fixes" each make a separate test pass but together regress a third. How do you attribute and resolve that, and what does it say about your test design?
The minority class has only a handful of examples. Beyond weighting the loss, what would you change in training and in evaluation so the reported metric is still trustworthy?
How would you turn this from a one-off script into a reproducible, monitored training job (versioning data, models, and metrics) for a team?