ML Concepts: Overfitting, Bias-Variance Trade-off, Class Imbalance, Missing Data

A machine learning knowledge round consisted of standard concept questions: overfitting and underfitting, the bias-variance trade-off, class imbalance, and how to handle missing data. Answer each part as you would aloud in the interview: define the concept precisely, explain how you would detect the problem in practice, and give concrete remedies together with their side effects. ### Clarifying Questions - Should the answers assume a particular model family, such as linear models, tree ensembles or neural networks, or cover several? - Is the setting classification, regression, or both? ### Part 1 — Overfitting, underfitting and the bias-variance trade-off Define overfitting and underfitting. How do you tell which of the two a model suffers from, and what do you do about each? Then explain the bias-variance trade-off and how it connects to the two failure modes. ```hint Compare two curves Put training error and validation error side by side, and watch how each one moves as model capacity or the amount of training data grows. ``` ```hint Split the error into parts Think of the expected test error of a regression model as a sum of separate terms, and ask which term each remedy you named actually reduces. ``` #### What This Part Should Cover - Definitions of both failure modes in terms of training and validation error - Diagnosis with learning curves or validation curves, using a held-out set or cross-validation - Remedies for each failure mode, with what each remedy costs - The bias-variance decomposition, and how capacity, regularization and data volume move each term ### Part 2 — Class imbalance One class makes up only a small fraction of the labeled data. What goes wrong if you train and evaluate the usual way, and how do you handle it? ```hint Start from the metric Ask what a model that always predicts the majority class would score on accuracy, and which metrics would expose it. ``` ```hint Data, loss or threshold Remedies can act on the training data, on the loss function, or on the decision threshold after training. Consider what each one does to the predicted probabilities. ``` #### Clarifying Questions for this Part - How rare is the minority class, and how many minority examples exist in absolute terms? - Are false positives and false negatives equally costly, and is the output used as a ranking, a yes/no decision or a probability? #### What This Part Should Cover - Why accuracy misleads, and which evaluation metrics stay informative - Resampling and class weighting, with their effects on calibration and overfitting - Choosing the decision threshold from the cost of each error type - Splitting and validating so that evaluation reflects the true class distribution ### Part 3 — Missing data Some features have missing values. How do you handle them? ```hint Why is it missing? Before choosing a fix, ask whether the chance that a value is missing depends on the value itself, on other observed features, or on nothing at all. ``` ```hint Where the fill-in values come from Whatever statistic or model you use to fill the gaps, think about which data it may be computed from, and what happens to it at prediction time. ``` #### Clarifying Questions for this Part - Are values missing in the same way at prediction time as in the training data? #### What This Part Should Cover - Missingness mechanisms and why they change the right treatment - The main options: dropping rows or features, simple and model-based imputation, missing-value indicators, and models that handle missing values natively - Avoiding leakage, and keeping training and serving consistent - How to check whether the chosen treatment helped ### What a Strong Answer Covers - Precise definitions tied to measurable quantities rather than intuition alone - A diagnostic step before each remedy - The trade-off of every remedy: bias introduced, calibration, information lost, compute - Awareness of leakage and of training/serving consistency across all three topics - Concrete examples from at least one model family ### Follow-up Questions - You increase regularization strength step by step, and validation error first falls and then rises. Explain both phases in bias-variance terms. - After oversampling the minority class, the model's predicted probabilities for that class are too high. Why, and how do you correct them? - A feature is missing far more often for one class than for the other. Is that a problem or a signal, and how do you treat it? - A very large neural network fits its training data perfectly yet still generalizes well. How does that square with the classical trade-off?