Design a **prompt playground** for developers and prompt engineers. The product lets users write prompts, choose model settings, run prompts against AI models, stream results back to the browser in real time, save prompt versions, compare outputs, collaborate with teammates, and inspect cost, latency, and safety issues. This is an open-ended system design problem. Drive it like a real interview: scope the requirements first, do back-of-the-envelope sizing, sketch a high-level architecture, then go deep on the hard parts (streaming, cancellation, tenant isolation, and cost control). State any assumptions you make explicitly. ### Constraints & Assumptions Assume the following scale unless you state different assumptions: - **100,000 monthly active users.** - **1 million prompt runs per day.** - Some runs **stream tokens** back to the browser in real time. - Users belong to **workspaces or organizations** (multi-tenant). - Prompt content and model outputs **may contain sensitive data**. Treat external model providers as a dependency you call over the network: calls cost real money per token, have variable latency, can be rate-limited or temporarily unavailable, and may refuse a request. ### Clarifying Questions to Ask A strong candidate scopes the problem before designing. Good questions to raise with the interviewer (these scope the whole system; per-Part clarifications appear under the relevant Part below): - What is the **read:write split**? How many runs vs. how many reads of history/usage dashboards, and how interactive (bursty) is traffic across the day and time zones? - What is the **average run duration and output length** (tokens), and what fraction of runs stream vs. fire-and-forget? This sizes concurrent connections and storage. - Are models **internal only**, external providers, or both — and do we need a fallback model when one is degraded? - What are the **latency targets** — especially time-to-first-token for streamed runs — and what durability guarantee do we owe a run if the browser disconnects mid-stream? - How strict is **tenant isolation and data retention** (e.g. per-workspace encryption, configurable deletion, regulatory deletion requests)? - Is **real-time multiplayer co-editing** in scope, or is versioning + comments sufficient for v1? --- The interviewer will expect you to cover the following. Treat each as a part of your answer. ### Part 1 — Core user flows and requirements Lay out the functional and non-functional requirements, the key user flows (author a prompt, run it, watch it stream, save a version, compare outputs, review history), and what you are explicitly leaving out of scope. ```hint Where to start Separate **functional** (what users do: author, configure, run, stream, version, compare, collaborate, inspect) from **non-functional** (latency, durability, tenant isolation, cost control, observability). Naming an explicit *out-of-scope* list is a strong signal. ``` #### What This Part Should Cover - **Functional vs. non-functional split** with the non-functional list anchored to *this* product's stakes (low time-to-first-token, durability of in-flight runs, tenant isolation, cost control). - An explicit **out-of-scope list** (e.g. training/fine-tuning, billing/payments, cursor-level co-editing) — naming what you are *not* building shows judgment. - **End-to-end user flows** that connect authoring → run → stream → save version → compare → review history, not just a feature list. ### Part 2 — APIs and data model Define the core entities and a sensible API surface. Pay attention to what must be preserved so a run is reproducible later, and where large/sensitive content lives. ```hint Data model Make prompt **versions immutable** and have each run *pin* the version plus a snapshot of the effective model config and variable values. Ask yourself what "reproducible" can and cannot mean when `temperature > 0`. ``` ```hint Storage split Multi-KB output bodies at 1M/day put different pressure on a store than small relational entities and permissions do. Should every field live in one store? Think about what each kind of data is queried by and how big it gets, and whether large bodies want to live somewhere they can be referenced by id rather than scanned. ``` #### What This Part Should Cover - A **clear entity model** (workspace, membership, prompt, immutable version, run, run output, quota, audit) with reproducibility built in: a run pins a version and snapshots the *effective* config + variable values. - A **storage split** argument — relational metadata vs. object-stored large bodies — justified by what each field is queried by and how big it gets. - A coherent **API surface** for authoring, running, streaming, comparing, and usage — including where reproducibility breaks down (nondeterminism at `temperature > 0`). ### Part 3 — High-level architecture Sketch the components and how a request flows through them. A good answer makes one structural decision very clear. ```hint The key split A playground is really **two systems under one UI**: a low-volume, consistency-first authoring/collaboration plane and a bursty, latency-sensitive execution plane. Drawing that boundary first organizes everything else. ``` ```hint Where the provider logic goes Consider a single internal **model gateway** in front of every model that owns provider-specific concerns (timeouts, retries, circuit breaking, token counting, error normalization) so the rest of the system stays provider-agnostic. ``` #### What This Part Should Cover - One **load-bearing structural decision** stated up front — separating the authoring/collaboration plane from the execution plane — and used to organize the rest. - A **request flow** through named components (gateway → orchestrator → queue → workers → model gateway → providers; a separate streaming tier) with each component's responsibility clear. - A **model gateway** that centralizes provider-specific concerns so the rest of the system stays provider-agnostic. ### Part 4 — Prompt execution, streaming, retries, and cancellation This is the core of the problem. Walk through the run lifecycle and be precise about how tokens stream to the browser, when you retry, and how cancellation actually stops billing. ```hint Streaming transport The request handler, the worker doing the generation, and the upstream provider are **three separate processes**. Think about a transport with built-in reconnect (e.g. SSE with `Last-Event-ID`) and whether the run should survive a browser that drops mid-stream. ``` ```hint Retries & idempotency `POST /runs` is the one place a client retry can **double-charge** a provider. What unique key prevents that? And which errors are safe to retry (timeouts, `429`, `5xx`) vs. which are deterministic and must be surfaced (`4xx`, content-policy refusals)? ``` ```hint Cancellation is the subtle one A cancel only saves money if it actually reaches the process talking to the provider and stops the upstream generation. If the browser tab is what closed, is that signal even reachable by the worker? Think about how cancel state travels to the worker, what the worker checks and when, and what happens if cancel and completion arrive at nearly the same moment. ``` #### What This Part Should Cover - A precise **run lifecycle** (`queued → running → succeeded/failed/canceled`) with the streaming path **decoupled** from the worker so a run survives a dropped browser. - A **retry policy** that distinguishes retryable (timeout, `429`, `5xx`) from deterministic errors and refusals, plus **idempotency** on `POST /runs` to prevent double-charging. - **Cancellation across process boundaries** — how cancel intent reaches the worker, what aborts the *upstream* generation to stop billing, and how the cancel-vs-completion race resolves. ### Part 5 — Versioning, collaboration, and run history Explain how editing produces versions, how teammates share and comment, and how run history is stored and queried at scale. ```hint Versioning + collaboration Immutable versions plus comments stored *separately* from versions keep discussion from mutating a version. For concurrent edits, decide whether you truly need OT/CRDT co-editing, or whether last-write-wins on a draft + visible version history is enough for v1 — and justify the cut. ``` ```hint History at scale Work out how fast the run table grows from the given 1M runs/day, and let that number drive the design. What does a table that size do to index pressure, retention, and heavy usage-dashboard aggregations? Think about how time-based access patterns and where the bodies live could shape the layout, and where you'd run the expensive aggregations. ``` #### What This Part Should Cover - A **versioning model** where editing produces immutable versions and comments live separately, so discussion never mutates a version. - A **justified scope cut** on concurrent editing (last-write-wins draft + version history vs. OT/CRDT), tied to whether live multiplayer is core. - **Run history at scale** driven by the 1M/day number: partitioning/retention, bodies in object storage, and heavy aggregations pushed off the transactional primary. ### Part 6 — Security, privacy, rate limiting, and abuse prevention Prompt content and outputs may be sensitive, so this part is load-bearing. Cover tenant isolation, encryption, secret handling, rate limits vs. spend quotas, and abuse prevention. ```hint Isolation & secrets Scope every query by a `workspace_id` derived from the **authenticated session**, never a client-supplied id. Where do external provider API keys live so workers never see them, and how do you keep sensitive prompt/output content out of logs? ``` ```hint Rate limits vs. quotas These are two distinct controls: **rate limits** (requests/sec — protect the system) and **spend quotas** (cost/runs per window — protect the budget). Where do you enforce each so a burst that slips past the edge still can't blow the budget? ``` #### What This Part Should Cover - **Tenant isolation** enforced server-side on every query via a session-derived `workspace_id`, plus encryption in transit/at rest and role-based access. - **Secret handling and log hygiene** — provider keys confined to the gateway, sensitive prompt/output content kept out of logs/traces. - **Two distinct controls** — rate limits (protect the system) and spend quotas (protect the budget) — enforced at the right layers, plus retention/deletion controls and abuse/anomaly detection. ### Part 7 — Observability, cost tracking, and scalability tradeoffs Describe what you measure to debug a slow or failed run, how you compute and store authoritative cost, and the major tradeoffs you are making (and why). ```hint What to measure Track queue depth/wait (early warning of provider slowdown), **time-to-first-token** and tokens/sec (user-perceived), error/refusal/timeout/cancel rates, and per-dimension cost. End-to-end tracing across gateway → queue → worker → provider makes a single slow run debuggable. ``` ```hint Cost tracking pitfall Compute usage authoritatively at run time (provider-reported tokens where available) and **store the cost on the run**. Why is recomputing cost from a hardcoded price constant at read time a trap? ``` ```hint Name the tradeoffs Be explicit: async queue vs. synchronous execution, relational + object-storage split, *conservative* caching of model outputs (tenant boundaries + nondeterminism), and graceful degradation — editing/history/compare should still work when the execution path or a provider is down. ``` #### What This Part Should Cover - The right **metrics and end-to-end tracing** to debug a single slow/failed run (queue wait, time-to-first-token, tokens/sec, error/refusal/timeout/cancel rates, per-dimension cost). - **Authoritative cost** computed at run time and stored on the run, with a clear reason why recomputing from a code constant at read time is a trap. - **Named tradeoffs** (async vs. sync, storage split, conservative output caching, graceful degradation) argued rather than asserted. ### What a Strong Answer Covers These dimensions span all parts; use them to judge the answer as a whole: - **Numbers drive design.** The capacity estimate (runs/sec average and peak, concurrent streams, storage growth, provider cost exposure) is actually used to justify decisions, not computed and dropped. - **The two-plane boundary is consistent.** The same authoring-vs-execution split shows up in the architecture, the data/storage layout, the scaling story, and the degradation story. - **Cost is treated as a first-class concern.** Idempotency, quotas, cancellation-that-stops-billing, and stored authoritative cost all reflect that every click spends real provider money. - **Tradeoffs are named and owned.** The candidate states what they are giving up (latency for durability, simplicity for isolation) and why, rather than presenting choices as free. ### Follow-up Questions Be ready for the interviewer to probe deeper: - How does the design change at **100× scale** (100M runs/day)? What breaks first — the queue, the streaming tier, the relational primary, or provider rate limits? - A provider becomes slow and starts timing out. Walk through exactly what users experience and how the system **degrades gracefully** instead of cascading. - A user wants to **re-run an old prompt version** and expects an identical output. What guarantee can you actually make, and why is bit-identical replay hard? - A run invokes a **tool with a side effect** (e.g. sends an email) and then a transient error occurs. How does your retry policy avoid sending the email twice?