Explain gradient descent and how you would respond when optimization appears stuck at a poor local solution. Distinguish local minima from saddle points, flat plateaus, ill-conditioning, and implementation errors. Recommend diagnostics and interventions for both convex and nonconvex objectives. ### Constraints & Assumptions - The objective is differentiable, but it may be nonconvex. - The answer should not claim that one optimizer guarantees the global minimum of a general neural network. - Include learning-rate, initialization, scaling, stochasticity, and numerical checks. ### Clarifying Questions to Ask - Is the loss convex in the parameters? - Are gradients finite and verified on a small example? - Does training loss stall, validation loss worsen, or both? - Is the plateau reproducible across seeds and optimizers? ```hint Diagnose before changing algorithms Plot loss and gradient norms, verify the implementation, and identify whether steps oscillate, vanish, or move slowly along poorly scaled directions. ``` ### What a Strong Answer Covers - The update rule and the role of step size. - Convex guarantees versus nonconvex limitations. - Saddle points, plateaus, conditioning, and noisy gradients. - Feature scaling, schedules, momentum, adaptive methods, restarts, and initialization. - Gradient checks, monitoring, and validation-based stopping. ### Follow-up Questions - Why can momentum accelerate progress in a narrow curved valley? - How does minibatch noise help escape some saddle regions? - When can Adam converge to a worse generalizing solution than SGD? - How would you detect exploding or vanishing gradients?