The output layer and final prediction

Convert the last token vector into vocabulary scores, then compare greedy decoding with temperature, top-k, and top-p sampling. These choices change how a token is selected; they do not establish whether a response is true.

After the prompt "Twinkle, twinkle, little" passes through a stack of transformer blocks, the model has a final vector at each input position. The output layer uses those vectors to make token predictions.

To continue the prompt, we need one more step: convert the last position’s vector into a score for each vocabulary token, then select the next token.

From hidden vectors to token scores

Separate this into two operations. The output layer produces scores and probabilities. The decoding strategy uses them to select a token.

For the next prediction, use the vector at the final token position. Causal attention allows that position to use the whole prefix, including the last input token. Earlier positions cannot use the later part of the prefix. This makes the final position the relevant one for continuation, but does not mean its vector preserves every detail of the input.