Explain gradient descent, learning rate, backpropagation, overfitting and batch size

Answer a set of machine learning fundamentals questions about how neural networks are trained. The interviewer covered gradient descent, the learning rate, backpropagation, underfitting and overfitting, and batch size; each Part below asks about one of these topics. Explain the mechanics and how you would act on them in practice, not only the definitions. ### Clarifying Questions - Should the answers assume a neural network trained on mini-batches with a differentiable loss, or a simpler model such as logistic regression? - How much math is expected: the update rule and the chain rule written out, or intuition first with formulas on request? - For batch size, is the setting a single GPU with a memory limit, or training distributed across several devices? ### Part 1 — Gradient descent and the learning rate Explain how gradient descent updates a model's parameters. What does the learning rate control, what happens when it is too large or too small, and how would you choose it in practice? ```hint Picture one step Take the simplest loss you can think of, with one parameter and a bowl shape, and work out where a single update lands for a few different step sizes. ``` #### What This Part Should Cover - The update rule and why it moves against the gradient, including the mini-batch version - What too large and too small learning rates do, and how each one looks in a loss curve - A practical way to pick the learning rate, and how it changes during training ### Part 2 — Backpropagation Explain how backpropagation computes the gradients that gradient descent needs. Walk through a small network: what the forward pass computes and stores, what the backward pass computes, and why this is far cheaper than computing each parameter's derivative separately. ```hint What flows backward When the chain rule is applied layer by layer, each layer needs one quantity from the layer above it. Identify that quantity, and what the forward pass must have saved to use it. ``` #### What This Part Should Cover - The chain rule applied layer by layer on a concrete small network - What must be stored from the forward pass, and what that means for memory - The cost of a backward pass compared with the alternatives, and why gradients vanish or explode in deep networks ### Part 3 — Underfitting and overfitting Define underfitting and overfitting, explain how you would tell which one a model has from its training and validation metrics, and say what you would do about each. ```hint Compare two curves Look at training error and validation error together: their levels, the gap between them, and how each changes as training continues or the model grows. ``` #### What This Part Should Cover - Definitions in terms of training and validation error, with bias and variance - A diagnosis procedure, including other causes of a gap between training and validation results - Remedies matched to each case ### Part 4 — Batch size How does the batch size affect training: the quality of the gradient estimate, speed per step and per epoch, memory, generalization, and the learning rate you should use? ```hint Noise has a price and a use A smaller batch gives a noisier gradient. Think about what that noise costs, what it might buy, and how the step size should change when the batch grows. ``` #### What This Part Should Cover - How the noise of the gradient estimate depends on the batch size - Hardware throughput and memory limits, and how to train with a batch larger than fits in memory - The interaction between batch size and learning rate, and the effect on generalization ### What a Strong Answer Covers - Correct update and chain-rule mechanics, with a small worked example rather than definitions alone - Symptoms tied to causes: loss curves for bad learning rates, training and validation gaps for underfitting and overfitting - Specific, practical remedies and tuning procedures for each problem - The links between topics: batch size and learning rate are tuned together, and gradient noise affects both optimization and generalization ### Follow-up Questions - Training loss suddenly becomes NaN after a few thousand steps. What do you check, and in what order? - Why does an adaptive optimizer such as Adam often need less learning-rate tuning than plain SGD, and when can it generalize worse? - Validation loss starts rising while validation accuracy keeps improving. How can both happen, and what do you do? - You double the number of GPUs, which doubles the global batch size. What do you change, and what do you watch?