During neural-network training, what happens if every weight is initialized to `0`? What happens if every weight is initialized to the same nonzero constant such as `1`? Explain why a good random initialization avoids those failures and how its scale should depend on layer width and activation choice. ### Constraints & Assumptions - Discuss a multilayer feed-forward network, not only a single linear model. - Distinguish weights from biases. - Address both symmetry and exploding or vanishing activation and gradient variance. ### Clarifying Questions to Ask - Which activation is used: sigmoid, tanh, ReLU, or another function? - Are normalization layers present? - Does the output layer have different initialization needs? ```hint Track two separate problems Breaking neuron symmetry and preserving signal scale are related goals, but one does not automatically guarantee the other. ``` ### What a Strong Answer Covers - Identical hidden units receive identical signals and updates under constant initialization. - Zero weights can block gradients to earlier layers even when an output-layer gradient exists. - Variance-aware initialization such as Xavier or He, tied to activation behavior. - Empirical checks on activation and gradient distributions. ### Follow-up Questions - Why can zero bias initialization be acceptable when zero weight initialization is not? - How do residual connections and normalization change the analysis? - What symptoms would an initialization problem create in training logs?