Design a Large-Scale ChatGPT-Style Assistant: Serving, Streaming and Scaling

Design ChatGPT: a large-scale conversational AI product in which users hold multi-turn conversations with a large language model. Explain how you would build and serve it, reasoning about scaling, latency and the serving infrastructure for the model. The report describes the question only as open-ended, so the scale, the feature set, the latency targets, and whether the model must also be trained or only served are open: settle them with the interviewer before designing. ```hint Where the time goes Think about which part of handling one message is orders of magnitude more expensive than everything else, and what the user experiences if they must wait for the whole reply. ``` ```hint The growing conversation Ask what the model needs to see on every turn of a long conversation, and what that costs as the conversation grows. ``` ### Clarifying Questions - How many daily and concurrent users are expected, and how many messages does a user send per day? - Do we serve an existing model on our own GPUs, call a third-party model API, or also train the model? - Must replies stream to the user as they are generated, and what targets apply to the first visible output and to the full reply? - Which features are in scope: conversation history, several models, file uploads, free and paid tiers? - What retention, privacy and content-safety rules apply to conversations? ### What a Strong Answer Covers - Requirements and estimates, with GPU capacity derived from token throughput rather than request counts - An API that delivers the reply incrementally, and how a dropped connection resumes - Conversation storage and how the context sent to the model is assembled within its context limit - Inference serving: request routing, batching, the memory held per active conversation, model replicas, queueing and admission control - Scaling and cost levers such as GPU pools, tiered rate limits, models of different sizes and prefix reuse - Reliability, content safety, abuse prevention and observability of latency, throughput and GPU utilisation ### Follow-up Questions - Traffic doubles at peak and more GPUs cannot be added quickly. What degrades first, and how? - A conversation grows longer than the model's context window. What do you send to the model? - A user's connection drops halfway through a reply. How does the client get the rest without regenerating it? - How would you roll out a new model version safely?