Hand-Compute a Small Neural Network Forward Pass with a Sigmoid Output

A multiple-choice question asks you to compute by hand the output of a small neural network for one input, where the output layer applies a sigmoid, and to give the result to three decimal places. The report does not give the network, so the one below is illustrative. It has two inputs, one hidden layer of two ReLU units (the hidden activation is an assumption; only the sigmoid output was reported) and one sigmoid output unit. ```text input: x1 = 1.0, x2 = 2.0 hidden h1: z1 = 0.5*x1 - 0.4*x2 + 0.1 h1 = ReLU(z1) = max(0, z1) hidden h2: z2 = -0.3*x1 + 0.8*x2 - 0.2 h2 = ReLU(z2) output: o = 1.2*h1 - 0.7*h2 + 0.5 y_hat = sigmoid(o) = 1 / (1 + exp(-o)) ``` Compute `y_hat` rounded to three decimal places, and the predicted class when the decision threshold is 0.5. ```hint Keep the activations straight Apply each activation to its own pre-activation before moving on to the next layer, and check the sign of each hidden pre-activation before applying ReLU. ``` ```hint Round once Carry full precision through every layer and round only the final probability. ``` ### Clarifying Questions - Is the hidden activation ReLU, sigmoid or something else? The answer depends on it. - Is the requested output the probability after the sigmoid, or the logit before it? - Should the answer be rounded to three decimals or truncated? ### What a Strong Answer Covers - Correct hidden pre-activations and activations, including a unit that ReLU switches off - The correct output logit, the sigmoid value to three decimals, and the predicted class - Rounding only at the end, with a way to check the exponential accurately enough for three decimals - The same computation in matrix form, which carries over to larger layers ### Follow-up Questions - Recompute the output with sigmoid hidden units instead of ReLU. How much does it change? - Write the forward pass in matrix form, with a weight matrix and a bias vector for each layer. - If the true label is 1 and the loss is binary cross-entropy, what are the gradients of the loss with respect to the output bias and the two output weights? - Why do frameworks compute the loss from the logit rather than from the probability?