Design a ChatGPT-style chat service with streaming model replies

Design ChatGPT. The report gives only this one-line prompt, so settle the scope with the interviewer first. In its usual form, users hold multi-turn conversations with a large language model in a web or mobile app: they send a message, watch the reply stream in token by token, and come back to their conversations later. ```hint Where the time and money go A reply takes seconds to produce and runs on scarce, expensive accelerator capacity. Decide how a request reaches that capacity, what happens when there is not enough of it, and what the user sees while the reply is produced. ``` ```hint Replies outlive connections Decide what is stored, and when, so that a dropped connection, a page refresh or a second device neither loses a reply nor shows it twice. ``` ### Constraints and Clarifications - Not given: number of users, traffic, model size, latency targets, and which features (file uploads, images, tools, sharing) are in scope. Ask, or state your assumptions. - Treat the language model as an existing inference service unless the interviewer wants model serving designed in depth. ### Clarifying Questions - Is the scope the product around an existing model (accounts, conversations, streaming, history), or also serving the model on accelerators? - How many daily users and messages, and what time to the first token of a reply is acceptable? - Are there free and paid tiers with different limits or priority, and are users limited in how much they can use? - Which features are in scope: conversation history and search, editing a message and regenerating a reply, file uploads, shared links, several devices per user? - What safety requirements apply to user inputs and model outputs? - What retention and deletion guarantees must conversations have? ### What a Strong Answer Covers - Scope and estimates that separate the chat product from model inference, sized in tokens per second and concurrent streams, not just requests per second - A streaming path from the model to the client, and what happens on disconnects, refreshes and a second device - A conversation and message data model, and how the prompt is assembled within the model's context window - Scheduling of scarce accelerator capacity: queueing, batching, admission control, per-tier rate limits and graceful degradation - Input and output moderation, abuse prevention, privacy and deletion - Failure handling, scaling across regions, and observability (time to first token, tokens per second, queue time) ### Follow-up Questions - A conversation grows longer than the model's context window. What do you send to the model? - Accelerator capacity drops by half during an incident. How does the service degrade, and who is served first? - A user closes the tab halfway through a reply and opens the conversation on a phone. What do they see? - How would you add tools, such as web search or running code, that the model can call in the middle of a reply?