Design a ChatGPT-like serving system

Q: Design a ChatGPT-like serving system

This question evaluates expertise in designing scalable machine-learning inference systems, covering chat-completion architecture, GPU capacity planning for large transformers, and stateful KV-cache design, layout, latency, and consistency considerations.

Q: How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

Question

Loading...

Design a ChatGPT-like system for inference serving.

Your design discussion should cover:

High-level architecture for chat completion (request routing, tokenization, model execution, streaming output, safety, observability).
A rough GPU count estimate to host a 400B-parameter transformer model using BF16 weights.
If you store the KV cache (attention keys/values for the prompt/history) in Redis , explain:
- what data is stored (granularity/layout),
- how inference workers read/write it during prefill and decode,
- latency/bandwidth implications and mitigations,
- failure modes and consistency choices.

Assume modern datacenter GPUs (e.g., 80GB class) and high-throughput networking. State any assumptions you make (context length, throughput targets, replication, etc.).

Design a ChatGPT-like serving system

Quick Overview

Solution

Comments (0)