# Design a Large-Scale Streaming AI Product Feature Design a product-facing AI feature for many concurrent users with streamed responses. Cover API and frontend streaming, queues, rate limits, caching and compute-cost controls, errors, and observability. ### Constraints & Assumptions - Generation can be slow or fail after partial output. - Clients may disconnect and retry. ### Clarifying Questions to Ask - Must a stream resume after reconnect? - Which requests are cacheable and what is the latency target? ### What a Strong Answer Covers - Admission, durable request state, SSE or WebSocket semantics, cancellation, budgets, and monitoring. ### Follow-up Questions - What changes at ten times traffic? - How do safety checks interact with streaming?