Explain Mixture of Experts and Expert Parallelism

Explain a mixture-of-experts transformer and how expert parallelism distributes its work. Trace what happens to a batch of tokens from routing through expert computation and combination. ### Constraints & Assumptions Use a sparse feed-forward MoE layer with top-k routing as the example. The number of experts, top-k value, capacity policy, and device placement are unspecified; state your choices when needed. ### Clarifying Questions Are all experts stored on one device or distributed? Can a token visit several experts? What happens when an expert receives more tokens than its capacity? Is the goal training throughput or serving latency? ### What a Strong Answer Covers Separate model architecture from device partitioning, explain sparse activation and routing communication, and identify load imbalance, capacity, and memory costs. ### Follow-up Questions Why can a model have many total parameters but comparatively few active parameters per token? What does a load-balancing loss encourage? When can communication erase the expected compute benefit?