Consider the BF16 layer `Y = GELU(XW + bias) + residual`, where `X` has shape `B x 4096` and `W` has shape `4096 x 4096`. The accelerator has 1 TB/s HBM bandwidth, 25 MB of SRAM, and peak BF16 throughput of 100 TFLOP/s. Analyze batch sizes `B = 1` and `B = 256`. Estimate compute time and HBM transfer time, then classify each case as primarily compute-bound or memory-bound. ### Constraints & Assumptions - Count the matrix multiplication as `2 * B * 4096^2` floating-point operations. - Treat a BF16 value as 2 bytes. - Assume the weights must be read once from HBM for each layer invocation and cannot all fit in the 25 MB SRAM. - Include one read of `X` and one write of `Y`; state how bias, residual, GELU, and cache effects would change the estimate rather than hiding them. ### Clarifying Questions to Ask - Are TB and MB decimal or binary units for this estimate? - Should the residual tensor be included as another HBM read? - Is peak throughput sustainable for these matrix shapes? ```hint Compare lower bounds Compute time is FLOPs divided by peak FLOP/s. Memory time is transferred bytes divided by bytes/s. ``` ### What a Strong Answer Covers - Correct dimensional arithmetic for weights, activations, FLOPs, and bytes. - A roofline comparison for both batches, with units and stated approximations. - Why batching reuses the same weights across more arithmetic and increases operational intensity. ### Follow-up Questions - What batch size is near the roofline crossover under the stated assumptions? - How would weight quantization change the classification? - Why might measured latency be worse than both lower bounds?