Microsoft Software Engineer Interview Experience — Sharding Embeddings and Language-Model Outputs

The author describes a Microsoft AI infrastructure virtual-onsite discussion centered on distributing large embedding tables and language-model output layers across GPUs. The interview compared splitting by vocabulary entries with splitting by hidden dimensions, asking how each choice affects memory use, communication, and connections to later computation.

Follow-ups examined routing token lookups, uneven traffic across shards, combining partial results, and selecting output candidates without immediately collecting every vocabulary score. The author characterized the session as practical distributed-systems reasoning for language models and found the questioning somewhat tricky. Interviewers repeatedly asked for justification of partition choices and identification of where communication occurs. The report does not describe other rounds or state a hiring outcome.