Explain What a Neural Network Is and How It Learns

The interviewer asked: "What is a neural network?" Give an explanation that a technical interviewer would accept as complete, covering what the model computes from its inputs and how it learns its parameters from data. ```hint Start from one unit Describe what a single unit computes from its inputs before stacking units into layers. ``` ```hint Stacking alone is not enough Consider what a stack of layers could represent if every layer computed only a weighted sum of its inputs. ``` ```hint Where the weights come from Explain how the weights get their values, starting from a measure of how wrong the network's outputs are. ``` ### Clarifying Questions - Does the interviewer want an intuitive explanation, the mathematical formulation, or both? - Should the answer stay with plain feed-forward networks, or also mention architectures such as convolutional, recurrent and transformer networks? - Should the explanation be tied to a particular kind of data or prediction task? ### What a Strong Answer Covers - What a unit and a layer compute, in terms of weights, biases and activation functions - Why non-linear activations are essential, and what depth adds - Training: a loss function, gradients computed by backpropagation, and an optimizer such as stochastic gradient descent - Generalization: overfitting, held-out validation data, and regularization - A small concrete example, and a balanced view of strengths and limitations compared with simpler models ### Follow-up Questions - Why does backpropagation cost only a small constant multiple of a forward pass, rather than one forward pass per weight? - Your dataset is small and noisy. What risks does a neural network bring, and what would you try first? - Why did deep networks with sigmoid activations train poorly, and what changed that? - On tabular data, neural networks are often matched or beaten by gradient-boosted trees. Why might that be?