# Design a Highly Available Conversational AI Service Design a conversational AI service that accepts a user message and streams a generated response. Begin with a deliberately minimal version that does not persist conversation history. Then evolve the design to support higher scale, stronger availability, fault tolerance, and useful operational metrics. State which later features require persistence rather than assuming them from the start. ### Constraints & Assumptions - Model inference is a separate dependency with variable latency and possible overload or failure. - Responses are streamed and clients may disconnect or retry. - The first version may treat each request independently. - Safety and authorization checks remain required even in the minimal version. - Capacity is finite; the system needs explicit overload behavior. ### Clarifying Questions to Ask - Is the first release text-only, and what request/response size limits apply? - What latency, availability, and geographic requirements matter? - Which safety, tenant isolation, and data-retention policies apply? - When conversation history is later added, is cross-device continuity required? ### Part 1: Minimal Stateless Path Describe the request flow, streaming protocol, validation, safety checks, inference call, cancellation, and error behavior without storing conversation history. #### What This Part Should Cover - Authentication, quotas, input limits, request IDs, and policy enforcement. - A gateway/orchestrator that invokes inference and relays bounded streaming output. - Cancellation propagation and clear treatment of retries after partial streams. ### Part 2: Scale, Availability, and Fault Tolerance Explain admission control, load balancing, dependency failure handling, regional design, and what changes when persistent conversations are introduced. #### What This Part Should Cover - Capacity-aware routing, queues with bounds, backpressure, and load shedding. - Timeouts, circuit breaking, retry limits, and no unsafe retry after unknown partial work. - Durable message ordering and idempotency only when history becomes a requirement. - Failure-domain isolation and graceful degradation. ### Part 3: Metrics and Verification Define service-level signals and tests that distinguish frontend streaming, orchestration, and model-provider problems. #### What This Part Should Cover - Time to first token, time to last token, generation rate, errors, cancellations, and queue delay. - Capacity, saturation, safety outcomes, dependency health, and cost indicators. - Load, fault-injection, disconnect, retry, and regional-failure tests. ### What a Strong Answer Covers - A genuinely simple first design that still has security, limits, and cancellation. - Explicit evolution from stateless requests to durable ordered conversations. - Backpressure and overload decisions rather than assuming infinite inference capacity. - Metrics tied to user experience and each internal failure boundary. ### Follow-up Questions - How would you resume a stream after a mobile connection drops? - Which data would you avoid persisting even after conversation history is added? - How would you separate model-quality regressions from serving-system regressions?