Explain Transformers and MoE in LLMs

You are interviewing for a role working with large language models (LLMs).

Explain the following concepts and how they relate to building and scaling LLMs:

1. Transformer architecture
   • What are the key components (e.g., self-attention, multi-head attention, positional encodings, feed-forward networks)?
   • How does the self-attention mechanism work at a high level?
   • Why are Transformers well-suited for language modeling compared to RNNs/LSTMs?
2. Mixture-of-Experts (MoE) architecture
   • What problem does MoE try to solve in the context of LLMs?
   • How does expert routing work conceptually (e.g., gating networks, top-k experts)?
   • What are the main trade-offs of MoE (compute efficiency vs. model complexity, training stability, load balancing)?
3. Collective communication and parallelism for LLMs
   • Briefly describe common forms of parallelism used to train and serve large models: data parallelism, tensor/model parallelism, and pipeline parallelism.
   • What is collective communication (e.g., all-reduce, all-gather, broadcast) and why is it critical for large-scale distributed training?
   • Give a simple example of where an all-reduce operation is used when training a Transformer model.

Focus on clear explanations that would help a strong software engineer understand how large language models are structured and scaled.