Compare Tensor Parallelism and Pipeline Parallelism

Compare tensor parallelism and pipeline parallelism for training or serving a transformer too large for one device. Explain what is partitioned, what communication each approach requires, and when you would consider combining them. ### Constraints & Assumptions No model size, device topology, or throughput target is specified. Use a transformer as the example and distinguish training from autoregressive inference. Avoid assuming one method is universally faster. ### Clarifying Questions Does the model fail to fit because of weights, optimizer state, or activations? What are the interconnect bandwidth, batch size, sequence length, and latency requirements? ### What a Strong Answer Covers Show a within-layer tensor split and a between-layer pipeline split, explain collectives and activation transfers, and discuss memory, utilization, and scheduling. ### Follow-up Questions What creates pipeline bubbles? How do microbatches help? Why might tensor parallelism be better confined to a fast local interconnect? How does a small inference batch affect the choice?