MLE Knowledge Collection

A comprehensive collection of common machine learning interview questions and detailed answers, organized by topic.

Table of Contents

1. ML Fundamentals
2. Regression
3. Regularization
4. Evaluation Metrics

---

ML Fundamentals

1. What are Overfitting and Underfitting?

Underfitting:

• Occurs when a machine learning model is too simple to capture the underlying patterns in the data
• Model performs poorly on both training and new unseen data
• Characterized by high training and validation errors
• Solutions:
• Use more complex models
• Add more relevant features
• Reduce regularization strength

Overfitting:

• Occurs when a model becomes too complex and starts memorizing training data instead of learning generalizable patterns
• Training error is significantly lower than validation error
• Model performs poorly on new, unseen data
• Solutions:
• Reduce model complexity
• Apply regularization techniques (L1, L2, dropout)
• Use cross-validation for model selection
• Collect more training data
• Apply data augmentation

2. What is the Bias-Variance Tradeoff?

Bias:

• The difference between predicted values and the expected value of real data
• Occurs when the model oversimplifies underlying patterns and makes strong assumptions
• Leads to underfitting where the model fails to capture true relationships between features and target variables

Variance:

• Measures how spread the predicted values are from the expected value
• High variance models are sensitive to specific data points and may memorize noise or outliers
• Leads to overfitting

The Tradeoff:

• Low variance models tend to be less complex with simple structure → can lead to high bias
• Low bias models tend to be more complex with flexible structure → can lead to high variance
• Decreasing one component often increases the other
• Goal: Find the right balance between bias and variance for optimal model performance

3. What are Common Methods to Prevent Overfitting?

1. Model Complexity Reduction

• Use simpler models
• Reduce the number of parameters

2. Regularization Techniques

• L1 regularization (Lasso)
• L2 regularization (Ridge)
• Dropout (for neural networks)
• Cross-validation for model selection

3. Early Stopping

• Stop training when validation performance stops improving

4. Data-based Approaches

• Collect more training data
• Data augmentation
• Remove noisy features

4. How to Determine if One Model is Better Than Another?

Given a set of ground truths and two models:

1. Evaluation Metrics

• Choose appropriate metrics based on the problem type
• Compare performance across multiple metrics

2. Cross-Validation

• Split data into multiple folds
• Train each model on different folds and test on alternating sets
• Evaluate average performance across all folds

3. Statistical Testing

• Hypothesis testing to determine if performance differences are statistically significant
• A/B testing in production environments

4. Domain Expertise

• Consider business requirements
• Evaluate model interpretability
• Assess computational efficiency

---

Regression

1. What are the Basic Assumptions of Linear Regression?

1. Linearity: There is a linear relationship between independent variables (X) and dependent variable (y)

2. Independence: No relationship or correlation between the errors (residuals) of different observations

3. Normality: The residuals are normally distributed

4. Homoscedasticity: The variability of errors (residuals) is constant across all levels of independent variables

5. No Multicollinearity: Independent variables are not highly correlated with each other

2. What Happens with Correlated Variables? How to Solve?

Problems with Correlated Variables:

• Unstable coefficient estimates
• Unreliable significance tests
• Difficulties interpreting individual variable contributions
• Inflated standard errors

Solutions:

• Feature selection (remove redundant features)
• Ridge regression (L2 regularization)
• Principal Component Analysis (PCA)
• Feature engineering to create uncorrelated features

3. Explain Regression Coefficients

• Coefficients represent the change in the dependent variable associated with a one-unit change in the corresponding independent variable, while holding other variables constant
• Interpretation example: If β₁ = 2.5, then a one-unit increase in X₁ leads to a 2.5-unit increase in y, assuming all other variables remain constant
• Important: Interpretation should be done with caution and within the context of the specific model and dataset

4. Relationship Between Minimizing Squared Error and Maximizing Likelihood

• In linear regression with Gaussian error assumptions, minimizing squared error is equivalent to maximizing the likelihood of observed data
• This connection arises because the squared error can be derived from the likelihood function assuming Gaussian errors
• When Gaussian error assumptions don't hold (e.g., non-Gaussian or heteroscedastic errors), this relationship may not be valid

5. How to Minimize Inter-correlation Between Variables?

1. Feature Selection: Remove highly correlated features
2. PCA: Transform features into uncorrelated principal components
3. Ridge Regression: Handles multicollinearity through L2 regularization
4. Feature Engineering: Create new uncorrelated features from existing ones

6. Can Linear Regression Handle Non-linear Relationships?

Simple linear regression cannot accurately capture non-linear relationships, but you can:

1. Add Interaction Terms: X₁ × X₂ to capture interaction effects
2. Polynomial Features: Add X², X³, etc.
3. Piecewise Linear Regression: Different linear models for different regions
4. Transform Variables: Log, square root, or other transformations
5. Switch to Non-linear Models: If relationship is strongly non-linear

7. Why Use Interaction Variables?

1. Capture Non-Additive Effects: When the effect of one variable depends on another
2. Improved Model Fit: Better representation of complex relationships
3. Context-Specific Relationships: Model how relationships change under different conditions
4. Avoid Omitted Variable Bias: Include important interaction effects
5. Enhanced Interpretability: Understand how variables interact

---

Regularization

1. L1 vs L2 Regularization: Differences

L1 Regularization (Lasso):

• Adds the sum of absolute values of parameters to loss function
• Formula: ||β||₁ = Σ|βᵢ|
• Can shrink coefficients to exactly zero
• Produces sparse models (feature selection)

L2 Regularization (Ridge):

• Adds the sum of squared parameters to loss function
• Formula: ||β||₂ = √(Σβᵢ²)
• Shrinks coefficients towards zero but not exactly zero
• Keeps all features but with reduced impact

2. Lasso Regression

• Full name: Least Absolute Shrinkage and Selection Operator
• Objective function: L = ||ŷ - y||₂ + λ||β||₁
• Where ŷ = f_β(x) is the prediction
• Can drive coefficients to exactly zero when λ is sufficiently large
• Useful for automatic feature selection
• Creates sparse models

3. Ridge Regression

• Linear regression with L2 regularization
• Objective function: L = ||ŷ - y||₂ + λ||β||₂
• Higher λ values result in more aggressive shrinkage
• All features retained but with reduced coefficients
• Handles multicollinearity well

4. Why is L1 Sparse but L2 is Not?

• Geometric interpretation:
• L1 norm creates diamond-shaped constraint regions with corners at zero
• L2 norm creates circular (ball-shaped) constraint regions
• The optimization solution often hits the corners of the L1 diamond (where coefficients are zero)
• For L2, the solution typically hits a point on the sphere where coefficients are non-zero
• L1 penalty is not differentiable at zero, creating a "pulling" effect towards exact zeros

5. Why Does Regularization Work?

• Adds constraints to the coefficient values
• Reduces model complexity by penalizing large coefficients
• Reduces variance at the cost of slightly increased bias
• Prevents overfitting by discouraging the model from fitting noise
• Handles multicollinearity by distributing weights among correlated features

6. Why Use L1/L2 Instead of L3/L4?

1. Mathematical Properties: L1 and L2 have well-studied properties that align with regularization goals
2. Computational Simplicity: Higher-order norms increase complexity without significant benefits
3. Interpretability: L1 (sparsity) and L2 (smoothness) have clear interpretations
4. Empirical Success: L1 and L2 have proven effective in practice
5. Optimization: Efficient algorithms exist for L1 and L2 regularization

---

Evaluation Metrics

1. Precision and Recall Trade-off

Precision:

• Measures how many positive predictions are actually true positives
• Formula: Precision = TP / (TP + FP)
• Focuses on the quality of positive predictions
• High precision = low false positives

Recall (Sensitivity):

• Measures how many actual positives are correctly identified
• Formula: Recall = TP / (TP + FN)
• Emphasizes completeness of positive predictions
• High recall = low false negatives

Trade-off:

• Improving one metric often decreases the other
• High precision, low recall: Conservative in predicting positives, few false positives but may miss true positives
• Low precision, high recall: Liberal in predicting positives, captures most true positives but generates more false positives
• Choice depends on the cost of false positives vs. false negatives

2. Metrics for Imbalanced Data

1. Precision and Recall: More informative than accuracy for imbalanced datasets
2. F1-Score: Harmonic mean of precision and recall, provides balanced evaluation
3. Area Under Precision-Recall Curve (AUPRC): Robust to class imbalance, focuses on positive class
4. ROC-AUC: Area under ROC curve, quantifies discriminative power
5. Matthews Correlation Coefficient (MCC): Considers all confusion matrix elements

3. Choosing Classification Metrics

Consider:

1. Problem understanding: Importance of correctly classifying each class
2. Class imbalance: Use appropriate metrics for imbalanced data
3. Business impact: Cost of false positives vs. false negatives
4. Domain knowledge: Industry-specific requirements
5. Multiple metrics: Often need to evaluate multiple aspects

4. Confusion Matrix

A table showing classification results:

• True Positives (TP): Correctly predicted positive cases
• True Negatives (TN): Correctly predicted negative cases
• False Positives (FP): Incorrectly predicted as positive
• False Negatives (FN): Incorrectly predicted as negative

From these, derive: Accuracy, Precision, Recall, F1-Score

5. TPR, FPR, and ROC

True Positive Rate (TPR):

• Also called Sensitivity or Recall
• TPR = TP / (TP + FN) = TP / (All Actual Positives)
• Measures classifier's ability to identify positive instances

False Positive Rate (FPR):

• FPR = FP / (FP + TN) = FP / (All Actual Negatives)
• Measures proportion of negatives incorrectly classified as positive

ROC Curve:

• Plots TPR vs. FPR at various classification thresholds
• Shows trade-off between sensitivity and specificity

6. AUC Interpretation

• Area Under the ROC Curve
• Represents probability that the model ranks a random positive instance higher than a random negative instance
• Range: 0 to 1
• AUC = 0.5: No better than random guessing
• AUC = 1.0: Perfect classifier
• AUC < 0.5: Worse than random (but can be inverted)
• Single number summary of model's discriminative ability

7. Ranking Metrics

Mean Reciprocal Rank (MRR):

• Formula: MRR = (1/m) × Σ(1/rankᵢ)
• Considers rank of first relevant item only
• Good when only one relevant result is expected

Recall@k:

• Formula: Recall@k = (# relevant items in top k) / (total # relevant items)
• Measures coverage of relevant items
• Challenge: Total relevant items can be very large

Precision@k:

• Formula: Precision@k = (# relevant items in top k) / k
• Measures precision of top k results
• Doesn't consider ranking quality within top k

Average Precision (AP):

• Computes average of precision@k for each relevant item
• Higher if relevant items appear earlier in list
• Considers both precision and ranking quality

Mean Average Precision (mAP):

• Average of AP across multiple queries
• Works well for binary relevance (relevant/not relevant)

Normalized Discounted Cumulative Gain (nDCG):

• DCG formula: DCGₚ = Σ(relᵢ / log₂(i+1))
• nDCG = DCG / IDCG (ideal DCG)
• Handles graded relevance scores (not just binary)
• Good for continuous relevance scores

8. Recommender System Metrics

1. Precision@k: Proportion of relevant items in top k recommendations
2. MRR: Good when expecting one relevant item
3. mAP: For binary relevance (liked/not liked)
4. nDCG: For graded relevance (ratings 1-5)
5. Diversity: Average pairwise dissimilarity between recommendations

• Low similarity score = high diversity
• Important for user engagement

Choosing between metrics:

• Binary relevance → mAP
• Graded relevance → nDCG
• Single relevant item → MRR
• User experience → Include diversity metrics

---

Note: This guide covers fundamental concepts commonly asked in machine learning interviews. Continue practicing with real problems and stay updated with the latest developments in the field.

How to Use This Page as a Prep Plan

Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.

For MLE Knowledge Collection, the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.

Video Walkthrough

This verified YouTube video gives a second pass on the same preparation area. Use it after reading the guide, then come back and turn the advice into a practice artifact.

FAQ

How much LeetCode should an MLE candidate do?

Do enough to communicate clearly under time pressure, but do not let generic algorithms crowd out ML fundamentals and system design.

What is the best way to review weak ML topics?

Use the interview feedback loop: miss a concept, write the explanation in your own words, then explain it aloud with one concrete example.

Should I prioritize ML system design or theory?

Prioritize the area most likely for the companies you are targeting, then keep a baseline in both so you can move between model quality and production constraints.