ChatGPT System Design

Interview exercise

Design a text assistant for a hypothetical service with persistent conversations, streamed answers, optional retrieval from an organization’s documents, and usage-based billing. Use the preceding lessons as a starting point. The exercise is about operating the service, not reconstructing OpenAI’s private architecture.

Spend about an hour on the design. Use the first ten minutes to agree on workload and user-visible behavior, twenty to trace the serving path, twenty to test failures, and the rest to explain the main capacity and quality trade-offs.

Define a request that can be recovered

Give each user turn a stable request identity and bind it to a conversation revision. Decide what happens if two turns arrive at the same time: serialize them, create branches, or reject a stale revision. The client should know which answer belongs to which turn.

Define accepted, generating, completed, canceled, and interrupted states. A streamed partial answer is useful to the user but is not a complete response. Explain whether an interrupted request can resume from saved generation state or must start a new attempt. If it starts again, avoid silently joining two potentially different generations.

Build the context before generating