Design a ChatGPT-like serving system
Company: Microsoft
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Technical Screen
Overview: This question evaluates expertise in designing scalable machine-learning inference systems, covering chat-completion architecture, GPU capacity planning for large transformers, and stateful KV-cache design, layout, latency, and consistency considerations.
Read the full Microsoft Software Engineer interview experience this question came from