Design an LLM-Powered Customer Service Chat System with Robust Failure Handling

Design a chat system for customer service. A customer opens a chat on a business's website or mobile app and talks to an AI agent whose replies are generated by a large language model (LLM) from an external LLM provider. The interviewer set the emphasis: - Walk through the happy path thoroughly, from the customer sending a message to the reply appearing on their screen. - Publish/subscribe (pub/sub) messaging is expected to be part of the design. - Error handling is the key part of the interview. For example: what happens if the LLM provider partially fails, and what happens if the network breaks? ### Constraints and Clarifications - The LLM provider is an external dependency outside your control: it can be slow, return errors, or fail partway through a response. - No traffic numbers or latency targets were given. Ask, or state your assumptions. ### Clarifying Questions - Is the agent fully automated, or can a conversation be handed off to a human support agent? - Are replies streamed to the customer as they are generated, or delivered as complete messages? - Which clients must be supported, and can one customer have the same conversation open on several devices or tabs? - Besides answering, can the agent take actions in the business's systems, such as looking up an order or issuing a refund? - How long must conversation history be kept, and who else reads it (analytics, quality review, a human agent console)? ### Part 1 — The happy path Walk through the system end to end for one customer message: the components involved, the protocol between client and servers, what is stored, and how the AI agent's reply is produced and delivered. ```hint Follow one message end to end Trace a single customer message from the moment it is sent until the reply is on screen, naming every component it passes through and what each one stores or acknowledges. ``` #### What This Part Should Cover - Client connection and protocol choice (for example WebSocket, server-sent events, long polling) - The conversation and message data model, identifiers and ordering - Orchestration of the LLM call: building context from history, calling the provider, streaming the reply - Persistence of messages and acknowledgement of delivery ### Part 2 — Pub/sub Where does publish/subscribe fit in your design? Describe the topics or channels, the publishers and the subscribers, and how a reply produced on one server reaches a customer whose connection is held by another server. ```hint Connections are pinned, work is not The server holding a customer's live connection is usually not the one that produced the reply. Decide how a message finds its way to the right connection, and which traffic must never be lost. ``` #### What This Part Should Cover - Topic or channel design, and which traffic is durable versus ephemeral - Decoupling connection servers from the agent workers - Delivery guarantees: at-least-once delivery, per-conversation ordering, deduplication - Fan-out to several devices and to other consumers ### Part 3 — Error handling This is the key part. Walk through what happens when (a) the LLM provider partially fails and (b) the network between the customer and your servers breaks. For each, describe what the system does, what the customer sees, and how everything recovers. ```hint Not every failure deserves a retry Sort failures by whether retrying can help and whether the first attempt may already have had an effect, then decide what to retry, at which layer, and how many times. ``` ```hint Both ends can resume After a dropped connection, think about what the client knows, what the server knows, and how they agree on which messages each side has already seen. ``` #### Clarifying Questions for this Part - What does "partially fails" mean here: a fraction of requests failing, slow or degraded responses, or a streamed reply that stops halfway? - Is it acceptable to show the customer a partial reply, or must replies be all or nothing? #### What This Part Should Cover - Classifying provider failures and choosing between retry, fallback and graceful degradation - Retry mechanics that do not overload a struggling provider (backoff, jitter, limits, circuit breaking) - Client-side handling: reconnection, idempotent resend, and visible pending and failed states - Recovering an interrupted streamed reply without duplicates or gaps ### What a Strong Answer Covers - A thorough happy path before any failure discussion - Explicit guarantees: no lost customer message, no duplicate replies, ordering within a conversation - Failure handling that spans the server and the client - Observability: provider error and latency metrics, per-conversation tracing, alerts - Trade-offs among latency, cost and reliability ### Follow-up Questions - The provider is completely down for an extended period. What does the customer experience, and what does your system do? - The customer sends three messages in quick succession while the agent is still answering the first. How do you handle it? - The agent can issue refunds. A retry after a timeout risks issuing one twice. How do you prevent that? - How would you test your failure handling before it meets production traffic?