ML Knowledge Collection V2

MLE Knowledge Compilation V2

Part 2: Loss Functions, Classification Models, Clustering, and Deep Learning

Table of Contents

1. Loss Functions
2. Logistic Regression
3. Support Vector Machines
4. Decision Trees
5. Clustering Algorithms
6. Deep Learning Fundamentals
7. Neural Network Architectures
8. Training Deep Networks

---

Loss Functions

1. Is Logistic Regression with MSE Loss Convex?

No, it is not convex.

• MSE loss with logistic regression creates a non-convex optimization landscape
• Can lead to multiple local minima
• This is why we use cross-entropy loss for logistic regression instead

2. Mean Squared Error (MSE)

Formula:

MSE = (1/N) × Σ(Yi - Ŷi)²

When to use MSE:

• Regression problems with normally distributed errors
• When you want to emphasize larger errors (due to squaring)
• When outliers should be heavily penalized
• Default choice for linear regression

Properties:

• Always non-negative
• Differentiable everywhere
• Sensitive to outliers
• Units are squared (e.g., dollars² if predicting prices)

3. Relationship Between Least Squares and MSE

• MSE is the objective function minimized by the Least Squares Method
• Least Squares finds coefficients that minimize the sum of squared residuals
• This is equivalent to minimizing MSE
• For linear regression: β* = argmin_β MSE(β)

4. KL Divergence (Relative Entropy)

Definition: Measures the difference between two probability distributions

Formula:

D_KL(P||Q) = Σ P(x) × log(P(x)/Q(x))

Interpretation:

• Expected value of log-ratio between P and Q, with respect to P
• Always non-negative (D_KL ≥ 0)
• D_KL = 0 if and only if P = Q
• Not symmetric: D_KL(P||Q) ≠ D_KL(Q||P)

Relationship to Cross-Entropy:

H(P,Q) = -Σ P(x) × log Q(x) = H(P) + D_KL(P||Q)

5. Logistic Regression Loss Function

Binary Cross-Entropy Loss:

L = -[y × log(p) + (1-y) × log(1-p)]

Where:

• y is the true label (0 or 1)
• p is the predicted probability

For all samples:

L = -(1/N) × Σ[yi × log(pi) + (1-yi) × log(1-pi)]

6. Logistic Regression Loss Derivation (MLE)

Maximum Likelihood Estimation approach:

1. Likelihood for single sample:

• P(y=1|x) = σ(wᵀx) = p
• P(y=0|x) = 1 - σ(wᵀx) = 1 - p
• Combined: P(y|x) = p^y × (1-p)^(1-y)

2. Log-likelihood for all samples:

LL = Σ[yi × log(pi) + (1-yi) × log(1-pi)]

3. Negative log-likelihood (our loss):

L = -LL = -Σ[yi × log(pi) + (1-yi) × log(1-pi)]

7. SVM Loss Function

Hinge Loss:

L = max(0, 1 - y × f(x))

Where:

• y ∈ {-1, +1} (class labels)
• f(x) = wᵀx + b (decision function)

Properties:

• Zero loss for correctly classified points with margin ≥ 1
• Linear penalty for margin violations
• Non-differentiable at the hinge point

Soft-Margin SVM Objective:

min (1/2)||w||² + C × Σ max(0, 1 - yi × f(xi))

8. Why Cross-Entropy for Multiclass Classification?

Reasons:

1. Natural extension of binary cross-entropy
2. Probabilistic interpretation via softmax
3. Maximum likelihood formulation
4. Well-behaved gradients for optimization
5. Handles multiple classes elegantly

Formula:

L = -Σ Σ yij × log(pij)

Where yij is 1 if sample i belongs to class j, else 0

9. Decision Tree Split Objectives

For Classification:

• Gini Impurity: Σ pi × (1 - pi)
• Entropy: -Σ pi × log(pi)
• Information Gain: Entropy(parent) - Weighted_Avg(Entropy(children))

For Regression:

• MSE: Minimize variance within nodes
• MAE: Minimize absolute deviations

10. Log Loss (Cross-Entropy Loss)

Definition: Measures performance by quantifying discrepancy between predicted probabilities and true labels

Binary Classification:

LogLoss = -[y × log(p) + (1-y) × log(1-p)]

When to use:

• Classification problems
• When you need probabilistic outputs
• When the model should be well-calibrated
• Default for logistic regression and neural networks

---

Logistic Regression

1. Logistic Regression vs SVM

Key Differences:

Optimization Methods:

• Logistic Regression: Gradient descent, Newton's method, L-BFGS
• SVM: Quadratic programming, SMO (Sequential Minimal Optimization)

---

Support Vector Machines

Key Concepts

1. Maximum Margin Classifier

• Finds hyperplane with largest margin between classes
• Support vectors: Points closest to decision boundary
• Only support vectors affect the decision boundary

2. Kernel Trick

• Maps data to higher dimensions implicitly
• Common kernels: Linear, RBF, Polynomial, Sigmoid
• Allows non-linear decision boundaries

3. Soft Margin (C parameter)

• Allows some misclassifications
• C controls trade-off between margin and errors
• High C: Less regularization, may overfit
• Low C: More regularization, may underfit

---

Decision Trees

1. How Trees Split Nodes

Regression Trees:

• Find split minimizing variance in child nodes
• Objective: min Σ(yi - ȳ_node)²
• Predictions: Mean of samples in leaf

Classification Trees:

• Find split maximizing class separation
• Metrics: Gini, Entropy, Information Gain
• Predictions: Majority class in leaf

2. Preventing Overfitting in Decision Trees

1. Pre-pruning (Early Stopping):

• Maximum depth limit
• Minimum samples for split
• Minimum samples in leaf
• Minimum impurity decrease

2. Post-pruning:

• Grow full tree, then remove branches
• Cost-complexity pruning
• Reduced error pruning

3. Ensemble Methods:

• Random Forests
• Gradient Boosting
• Reduces variance through averaging

4. Feature Selection:

• Use only relevant features
• Random feature subsets (Random Forest)

5. Cross-Validation:

• Select optimal hyperparameters
• Avoid overfitting to validation set

---

Clustering Algorithms

1. K-Means Clustering

Algorithm Steps:

1. Initialize k cluster centroids randomly
2. Assign each point to nearest centroid
3. Update centroids as mean of assigned points
4. Repeat until convergence

Objective Function:

WCSS = Σ Σ ||xi - μj||²

(Within-Cluster Sum of Squares)

Convergence:

• Always converges (WCSS monotonically decreases)
• May converge to local optimum
• Solution depends on initialization

Stopping Criteria:

• No change in assignments
• Small change in centroids
• Maximum iterations reached
• WCSS improvement below threshold

Choosing k:

• Elbow Method: Plot WCSS vs k, find "elbow"
• Silhouette Score: Measure cluster quality
• Gap Statistic: Compare to random data
• Domain Knowledge: Business requirements

2. Expectation-Maximization (EM) Algorithm

Purpose: Estimate parameters with latent/missing variables

Algorithm:

1. Initialize parameters θ
2. E-step: Compute expected value of latent variables given current θ
3. M-step: Update θ to maximize expected log-likelihood
4. Repeat until convergence

Properties:

• Guarantees non-decreasing likelihood
• Converges to local optimum
• Sensitive to initialization

Applications:

• Gaussian Mixture Models
• Hidden Markov Models
• Missing data imputation
• Topic modeling

3. Gaussian Mixture Models (GMM)

Model:

p(x) = Σ πk × N(x|μk, Σk)

Where:

• πk: Component weights (Σπk = 1)
• μk: Component means
• Σk: Component covariances

GMM vs K-Means:

---

Deep Learning Fundamentals

1. Neural Network Basics

Key Components:

• Neurons: Basic computational units
• Layers: Input, Hidden, Output
• Weights & Biases: Learnable parameters
• Activation Functions: Non-linear transformations

Forward Propagation:

z = Wx + b
a = f(z)

Where f is activation function

2. Activation Functions

Common Choices:

1. ReLU (Rectified Linear Unit):

• f(x) = max(0, x)
• Pros: Simple, efficient, avoids vanishing gradient
• Cons: Dead neurons, not zero-centered

2. Sigmoid:

• f(x) = 1/(1 + e^(-x))
• Pros: Smooth, probabilistic interpretation
• Cons: Vanishing gradient, not zero-centered

3. Tanh:

• f(x) = (e^x - e^(-x))/(e^x + e^(-x))
• Pros: Zero-centered
• Cons: Vanishing gradient

4. Leaky ReLU:

• f(x) = max(αx, x), α small
• Pros: Avoids dead neurons
• Cons: Additional hyperparameter

5. GELU (Gaussian Error Linear Unit):

• Modern choice for transformers
• Smooth approximation of ReLU

3. Backpropagation

Chain Rule Application:

∂L/∂w = ∂L/∂a × ∂a/∂z × ∂z/∂w

Key Steps:

1. Compute loss at output
2. Propagate gradients backward
3. Update weights using gradients

4. Gradient Descent Variants

1. Batch Gradient Descent:

• Use entire dataset
• Stable but slow

2. Stochastic (SGD):

• One sample at a time
• Noisy but can escape local minima

3. Mini-batch:

• Subset of data
• Balance between batch and SGD

4. Momentum:

v = βv + (1-β)∇L
w = w - αv

5. Adam:

• Adaptive learning rates
• Combines momentum and RMSprop
• Most popular in practice

---

Neural Network Architectures

1. Convolutional Neural Networks (CNNs)

Key Components:

• Convolutional Layers: Feature extraction with filters
• Pooling Layers: Downsample spatial dimensions
• Stride & Padding: Control output dimensions

Why CNNs for Images:

• Parameter sharing (same filter across image)
• Translation invariance
• Hierarchical feature learning
• Reduced parameters vs fully connected

2. Recurrent Neural Networks (RNNs)

Types:

• Vanilla RNN: Simple but suffers from vanishing gradient
• LSTM: Long Short-Term Memory, gates control information flow
• GRU: Gated Recurrent Unit, simplified LSTM

Applications:

• Sequential data (text, time series)
• Variable length inputs
• Memory of previous states

3. Transformers

Key Innovation: Self-attention mechanism

Components:

• Multi-head Attention: Attend to different positions
• Positional Encoding: Inject sequence order
• Feed-forward Networks: Process attended features

Advantages:

• Parallelizable (unlike RNNs)
• Long-range dependencies
• State-of-the-art for NLP

---

Training Deep Networks

1. Common Challenges

Vanishing/Exploding Gradients:

• Causes: Deep networks, poor initialization, activation functions
• Solutions:
• Better initialization (Xavier, He)
• Batch normalization
• Residual connections
• Gradient clipping

Overfitting:

• Solutions:
• Dropout
• L1/L2 regularization
• Data augmentation
• Early stopping
• Ensemble methods

2. Regularization Techniques

1. Dropout:

• Randomly set neurons to 0 during training
• Prevents co-adaptation
• Typically 0.2-0.5 dropout rate

2. Batch Normalization:

• Normalize inputs to each layer
• Reduces internal covariate shift
• Acts as regularizer

3. Layer Normalization:

• Normalize across features
• Better for RNNs/Transformers

4. Weight Decay:

• L2 penalty on weights
• Prevents weights from growing too large

3. Advanced Training Techniques

Transfer Learning:

• Use pre-trained models
• Fine-tune on specific task
• Especially useful with limited data

Learning Rate Scheduling:

• Decay over time
• Warm-up for stability
• Cosine annealing
• Reduce on plateau

Mixed Precision Training:

• Use FP16 for speed
• Keep master weights in FP32
• Gradient scaling for stability

Gradient Accumulation:

• Simulate larger batches
• Useful for memory constraints

---

Interview Tips

Common Deep Learning Questions

1. "How do you handle imbalanced datasets in deep learning?"

• Weighted loss functions
• Oversampling/undersampling
• Focal loss
• Class-balanced sampling

2. "Explain batch normalization"

• Normalizes inputs per mini-batch
• Learnable scale and shift parameters
• Helps with gradient flow
• Different behavior train/test

3. "When would you use CNN vs RNN vs Transformer?"

• CNN: Spatial data, local patterns
• RNN: Sequential data, online processing
• Transformer: Long sequences, parallelization needed

4. "How do you debug a neural network that's not learning?"

• Check data pipeline
• Verify loss calculation
• Start with simple model
• Monitor gradients
• Reduce learning rate
• Check for numerical issues

Practical Considerations

1. Model Selection:

• Start simple, add complexity
• Consider computational constraints
• Think about deployment requirements

2. Hyperparameter Tuning:

• Learning rate most important
• Use systematic search (grid, random, Bayesian)
• Monitor validation metrics

3. Production Considerations:

• Model size and inference speed
• Quantization for deployment
• A/B testing framework
• Model versioning and monitoring

---

Note: This compilation covers advanced topics commonly asked in MLE interviews. Make sure to understand both theoretical concepts and practical implementations.

How to Use This Page as a Prep Plan

Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.

For ML Knowledge Collection V2, the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.

Video Walkthrough

This verified YouTube video gives a second pass on the same preparation area. Use it after reading the guide, then come back and turn the advice into a practice artifact.

FAQ

How should I use this guide?

Read it once for the structure, then turn each section into a practice task with a visible artifact.

What should I do if I am short on time?

Prioritize the skills most likely to be tested, then do one mock or timed drill to expose the largest gap.

How do I know I am ready?

You can explain your approach clearly, recover from hints, and name tradeoffs without relying on memorized wording.