The feedforward network and final assembly

See how a feedforward network transforms each token vector, then combine it with attention, residual connections, and normalization to form a transformer block.

Multi-head attention mixes information from the positions that each token is allowed to attend to. Its output is another matrix of token vectors, now updated using that context.

The next operation transforms the features within each of those vectors. Unlike attention, it does not mix information between token positions. This distinction explains why a transformer block includes both operations.

This is the job of the final core component in our block: the feedforward network (FFN).

Processing each token vector

The FFN applies the same small neural network separately to every token vector. The weights are shared across positions within a block; each token does not have its own separately trained network. Attention mixes information across permitted positions, while the FFN transforms each position’s features.