Compare Mixture-of-Experts and Dense Transformers

# Compare Mixture-of-Experts and Dense Transformers Compare mixture-of-experts and dense Transformer models. Explain capacity, compute, routing, memory, communication, training stability, and serving trade-offs rather than naming one architecture universally better. ### Constraints & Assumptions - Compare models under a stated active-compute and quality target. - An MoE layer activates only a subset of experts per token but may store many experts. - Expert placement and communication are part of the architecture cost. ### Clarifying Questions to Ask - Is the constraint training compute, serving latency, memory, or model quality? - How many experts are active per token? - What hardware topology carries all-to-all traffic? ```hint Separate total from active parameters State both stored model capacity and the parameters or experts used for one token. ``` ### What a Strong Answer Covers - Dense baseline and sparse routing mechanism. - Capacity-per-compute advantage and specialization potential. - Load balancing, dropped tokens, routing collapse, and communication. - Serving memory, batching, latency variance, reliability, and operational complexity. ### Follow-up Questions 1. How would expert imbalance appear in production metrics? 2. When can a smaller dense model be the better product choice?