Shard an LM Head and Combine Distributed Logits

An LM head maps a hidden state through a weight matrix `[H,V]` to vocabulary logits. Compare sharding the weight by vocabulary and by hidden dimension. Explain what each rank computes, which reductions are required, and whether full-vocabulary logits must be gathered before top-k selection or sampling. ### Constraints & Assumptions State the incoming hidden-state layout and the requested output operation. Exact full-distribution sampling is different from top-k sampling; do not silently replace one with the other. ### Clarifying Questions Is the hidden vector replicated or sharded? Does the consumer need all logits, only the global top k, or an exact categorical sample? What output layout can downstream operations consume? ### What a Strong Answer Covers Complete versus partial dot products, all-reduce/reduce-scatter or layout conversion, local candidates and global top-k merging, and distributed normalization when needed. ### Follow-up Questions Why can local top-k candidates suffice for exact global top k? Why are they insufficient for unrestricted full-softmax sampling? Where does communication move if an earlier layer already outputs hidden shards?