Design a Chat Assistant Inference Service: Streaming, Free-Tier Limits, GPU Scheduling

Design the serving system behind a ChatGPT-style chat assistant: users send messages in a conversation, and a large language model running on GPUs generates the reply token by token. Focus on four areas: **streaming** the reply to the user as it is generated, **rate limiting** for a free tier alongside paid tiers, **scheduling requests onto GPUs**, and **latency**. ```hint Two latencies, not one Separate the time until the first word appears from the pace at which the rest of the reply arrives, and ask which parts of the system control each. ``` ```hint The GPU's scarce resource Think about what limits how many conversations one GPU server can generate for at once, and how the scheduler should admit, batch and evict work around that limit. ``` ### Clarifying Questions - What scale is expected: daily users, peak concurrent generations, typical prompt and reply lengths? - What latency targets apply, for the first token and for the tokens after it, and do they differ by tier? - How should the free tier be limited: by messages, by tokens, by time window, or by access to certain models? - Are there several models of different sizes, and can a request fall back to a smaller model under load? - How long is the conversation context, and is conversation history stored server-side? ### What a Strong Answer Covers - The request path from client to model and back, with a streaming protocol and how it copes with disconnects and retries - Latency broken into its components, with targets and where each is optimized - GPU serving: batching of concurrent requests, memory for the attention key-value cache, separating prompt processing from token generation, and routing across replicas - A rate-limiting design for the free tier: the limits, where they are enforced, how they scale, and behavior under overload - Capacity estimates, autoscaling limits for GPUs, graceful degradation, and observability ### Follow-up Questions - A user's connection drops halfway through a long reply. What happens to the generation, and what does the user see on reconnect? - How would you reuse computation for users who share the same long system prompt? - How would you roll out a new model version with no drop in quality or latency? - During a traffic spike, which users get degraded first, and how?