# Prevent Vanishing Gradients in Deep Networks ### Clarifying Questions to Ask - Are we discussing feed-forward networks, recurrent networks, or both? - Should the answer focus on mathematical causes, practical architecture choices, or debugging evidence? ### Part 1: Regularization Explain why regularization is used during training. Distinguish the generalization problem it addresses from optimization problems such as vanishing gradients. **Candidate hint:** State what behavior regularization discourages and how that changes train-versus-validation performance. #### What This Part Should Cover - The objective of regularization and at least two common mechanisms - The difference between controlling overfitting and improving gradient flow - A trade-off introduced by excessive regularization ### Part 2: Activation Functions Compare activation functions with respect to gradient propagation. Which choices are generally less prone to saturation, and what limitations do they still have? **Candidate hint:** Reason from the derivative over the input range rather than from the activation's name. #### What This Part Should Cover - Saturation and derivative behavior - Practical alternatives and their failure modes - The interaction with initialization and activation scale ### Part 3: Architecture and Skip Connections Explain architectural choices that help train very deep networks, including how residual or skip connections affect the backward pass. **Candidate hint:** Write the derivative of a residual block in symbolic form and inspect the additional path. #### What This Part Should Cover - A correct gradient-path argument - Depth, initialization, and signal-preserving architecture choices - Limits: skip connections help but do not guarantee stable training ### Part 4: Normalization Explain how batch normalization can affect optimization and gradient flow. Include at least one situation where it is unsuitable or behaves differently between training and inference. **Candidate hint:** Discuss activation distributions, learned scale and shift, and the source of inference statistics. #### What This Part Should Cover - The training transformation and running-statistic behavior - Optimization benefits without claiming that normalization solves every gradient problem - Batch-size, sequence-model, or distribution-shift caveats ### What a Strong Answer Covers - Clear separation of generalization tools from gradient-flow tools - Mathematical intuition tied to practical design and diagnosis - Interactions among activation, initialization, normalization, and architecture - Appropriate caveats instead of one universal prescription ### Follow-up Questions 1. How would you diagnose vanishing gradients from training telemetry? 2. Why can a well-chosen initialization help at the beginning but not guarantee stability later? 3. How do pre-normalization and post-normalization residual blocks differ? 4. What changes for a recurrent network trained over long sequences?