Train vs Test Error Diagnosis, Overfitting Fixes by Model Type, and Cross-Validation

In a technical screen for a machine learning engineer role, the first part of the interview is a fundamentals discussion with three connected questions: what training and test error reveal about a model, how to fix overfitting in different kinds of models, and how to set up data splits for validation. ### Clarifying Questions - What metric is the error, and what would a trivial baseline (predicting the mean or the majority class) score? Without a reference point, "high" and "low" are only relative. - Were the training and test sets drawn from the same population and time period, and how was the split made? - Is the training error computed with the final model in inference mode, or reported while training was running? - How large is the test set? ### Part 1 — Reading training and test error The interviewer shows five pairs of (training error, test error). They include these patterns: 1. Low training error, high test error 2. High training error, low test error 3. Both low 4. Both high For each pattern, explain the likely causes and what you would check or change next. ```hint Level and gap Look separately at how high the training error is and at how far apart the two errors are; each points to a different kind of problem. ``` ```hint When unseen data looks easier If a model does better on data it never saw than on the data it was fit to, compare the two datasets themselves, not only the model. ``` #### What This Part Should Cover - Bias and variance read from the level of the training error and the size of the gap - Causes beyond model capacity: differences between the two sets, noise, leakage, measurement artifacts - Concrete checks that tell these causes apart - The next action for each pattern ### Part 2 — Fixing overfitting by model family How do you reduce overfitting in a linear model, a tree-based model, and a neural network? Give remedies specific to each family, not only generic ones. ```hint Find the capacity knob For each family, identify what lets the model fit noise, and the parameter or procedure that limits it. ``` #### What This Part Should Cover - Remedies for linear models - Remedies for single trees and for tree ensembles, which behave differently - Remedies for neural networks - How the strength of each remedy is chosen without touching the test set ### Part 3 — Train, validation and test splits, and k-fold cross-validation How do you divide data into training, validation and test sets, and how do you run k-fold cross-validation? ```hint One job per split Decide what each set is allowed to influence, then check that nothing from the held-out data can reach training, including through preprocessing. ``` #### Clarifying Questions for this Part - Is the data ordered in time, and can the same entity (for example, the same user) appear in many rows? - Roughly how much data is there? #### What This Part Should Cover - The role and rough size of each split - The k-fold procedure step by step, including which model is trained at the end - Split variants for imbalanced, grouped and time-ordered data - Leakage through preprocessing or duplicate rows ### What a Strong Answer Covers - A clear diagnostic framework (achievable error, training error, gap) rather than a list of memorized cases - Recognition that high training error with low test error points to the data or the measurement, not to the model's capacity - Family-specific overfitting remedies, named with the hyperparameters that implement them - A validation method that keeps the test set for a single final estimate - Attention to the structure of the data (time order, groups, class balance) when splitting ### Follow-up Questions - Your cross-validation score is consistently better than your test score. What could cause that? - When would you prefer a single validation split over k-fold cross-validation? - How do you tune hyperparameters and still report an unbiased estimate of performance? - With gradient-boosted trees, you stop training early based on the validation set. How does that change what the validation score means?