Explain learning-rate fluctuation and vanishing gradients
Company: Pinterest
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: easy
Interview Round: Technical Screen
### ML Fundamentals
Answer the following conceptual questions:
1. **Learning rate vs. training stability**: Why can training metrics (loss/accuracy) **fluctuate or oscillate** when the learning rate is too large? What happens when it is too small?
2. **Vanishing gradients in fully connected networks**: In a deep fully connected network trained with backpropagation, are **vanishing gradients** more likely to affect layers closer to the **input** or closer to the **output**? Explain why, and name common mitigations.
Overview: This question evaluates a candidate's understanding of optimization dynamics (effects of learning rate on training stability) and backpropagation-related issues (vanishing gradients) in deep neural networks, assessing competencies in training behavior and gradient propagation within the Machine Learning domain.
Community answers
Answer by swemleds
Learning Rate is multiplied to the derivative of loss function, gradient and subtracted to the original weight. So if it is too large, the change in weights at each update will be too large and fluctuate. If it is too small, the training gets too slow with small parameter update.
Layers closer to input because the change driven by gradient gets smaller through back propagation from output to input layer by chain rule. It can be mitigated through residual connections, ReLU activation, or normalization.