Design a ChatGPT-like serving system
Company: Microsoft
Role: Software Engineer
Category: System Design
Difficulty: nan
Interview Round: Technical Screen
Design a ChatGPT-like system for inference serving.
Your design discussion should cover:
1. **High-level architecture** for chat completion (request routing, tokenization, model execution, streaming output, safety, observability).
2. A **rough GPU count estimate** to host a **400B-parameter** transformer model using **BF16** weights.
3. If you store the **KV cache** (attention keys/values for the prompt/history) in **Redis**, explain:
- what data is stored (granularity/layout),
- how inference workers read/write it during prefill and decode,
- latency/bandwidth implications and mitigations,
- failure modes and consistency choices.
Assume modern datacenter GPUs (e.g., 80GB class) and high-throughput networking. State any assumptions you make (context length, throughput targets, replication, etc.).
Quick Answer: This question evaluates expertise in designing scalable machine-learning inference systems, covering chat-completion architecture, GPU capacity planning for large transformers, and stateful KV-cache design, layout, latency, and consistency considerations.