# Explain and Compare SGD and Adam Explain how stochastic gradient descent updates model parameters and how Adam modifies those updates using moving estimates of first and second gradient moments. Derive the core update equations, explain bias correction, and compare the optimizers on convergence behavior, tuning, memory, and generalization. ### Constraints & Assumptions - Use mini-batch gradients rather than full-dataset gradients. - Include learning rate, Adam's two decay coefficients, numerical stabilizer, and time step. - Distinguish momentum SGD from plain SGD when making comparisons. - Do not claim one optimizer is universally superior. ### Clarifying Questions to Ask - Is the comparison about early training speed, final validation quality, or both? - Does weight decay mean an L2 gradient term or decoupled weight decay? - Are gradients sparse or highly different in scale across parameters? ```hint Track state per parameter Adam stores both a moving mean and a moving uncentered second moment of gradients. ``` ```hint Inspect the first steps Zero initialization pulls both estimates toward zero, which bias correction compensates for. ``` ### What a Strong Answer Covers - Correct update equations and the role of each state variable. - Why Adam's zero-initialized moments need bias correction early in training. - Learning-rate scheduling, memory, scale adaptation, and generalization trade-offs. - Failure modes such as unstable steps, stale moment estimates, and coupled weight decay confusion. ### Follow-up Questions - How does AdamW differ from adding an L2 term to Adam's gradient? - Why might a practitioner switch from Adam to SGD late in training?