This question evaluates system design skills for ML-powered chatbots with a front-end focus, testing understanding of client-side privacy and storage lifetimes, streamed response delivery, stateless relay architectures, authentication boundaries, rate limiting, and failure-mode handling.
##### Question
Design an AI chatbot system with a front-end focus, under the following constraints:
1. User messages and conversation history are stored only in the user's browser — never in any server-side database.
2. Bot responses must be streamed to the user (tokens appear incrementally).
3. Refreshing the page starts a brand-new conversation (no persisted history).
4. User credential / authentication handling must be addressed on the client without leaking provider secrets.
In your design, cover:
- The overall architecture (client SPA, edge/relay layer, backend LLM provider) and the end-to-end data flow.
- How streaming is implemented, and the trade-offs of Server-Sent Events (SSE) vs. WebSockets.
- Session management without a server database (where conversation state and tokens live, and how refresh resets them).
- Security for user credentials and provider API keys (OAuth/OIDC + PKCE, token lifetimes, scopes, CORS/CSP/XSS).
- Rate limiting and abuse prevention when you have no durable server-side store.
- Failure modes and retry strategy (network drops mid-stream, 429s, 5xx, timeouts, token expiry, oversized context).
- The trade-offs of the no-database approach (privacy and cost vs. lost continuity, larger per-request payloads, weaker analytics/personalization).
Quick Answer: This question evaluates system design skills for ML-powered chatbots with a front-end focus, testing understanding of client-side privacy and storage lifetimes, streamed response delivery, stateless relay architectures, authentication boundaries, rate limiting, and failure-mode handling.
mediumSoftware EngineerTechnical ScreenML System Design
14
0
Question
Design an AI chatbot system with a front-end focus, under the following hard constraints:
User messages and conversation history are stored
only in the user's browser
— never in any server-side database.
Bot responses must be
streamed
to the user (tokens appear incrementally as they are generated).
Refreshing the page starts a brand-new conversation
— there is no persisted history.
User credential / authentication handling must be addressed
on the client without leaking provider secrets
(the LLM provider API key must never reach the browser).
Walk through your end-to-end design. At minimum, address: the overall architecture and data flow (client SPA, edge/relay layer, backend LLM provider); how streaming is implemented and the trade-off between Server-Sent Events (SSE) and WebSockets; session management without a server database (where conversation state and tokens live, and why a refresh resets them); security for both user credentials and the provider API key (OAuth/OIDC + PKCE, token lifetimes, scopes, CORS/CSP/XSS); rate limiting and abuse prevention with no durable server-side store; failure modes and a retry strategy (mid-stream network drops, 429s, 5xx, timeouts, token expiry, oversized context); and the trade-offs of going database-free (privacy and operational cost vs. lost continuity, larger per-request payloads, and weaker analytics/personalization).
Constraints & Assumptions
No server-side storage of conversation content
of any kind: no database, and (be careful) no request/response
body logging
at the app, CDN, or WAF layer either — otherwise the guarantee is silently violated.
Refresh / tab-close = fresh conversation.
No cross-device or cross-session continuity is expected.
The
provider API key is a shared secret
the relay holds (assume
we
pay for tokens via one key). Note where a "bring-your-own-key" variant changes the design.
Assume a single streaming chat-completion call per turn (note where tool-calling / RAG would change the transport choice).
The constraint forbids storing conversation
content
— it does
not
forbid tiny, content-free operational counters (e.g. short-TTL rate-limit buckets) that hold no message text.
HTTPS everywhere; a public browser client (no client secret can be kept).
Clarifying Questions to Ask
Who pays for tokens?
A shared provider key behind the relay (the common case), or does each user bring their own key?
Single LLM call, or tools / RAG / multi-step agents?
This changes whether a unidirectional stream is sufficient or a duplex channel is warranted.
What exactly does "no server DB for messages" forbid?
Does it also forbid content-free operational counters (rate-limit buckets), or only conversation content?
What is the provider's data-retention policy?
"No server DB" only holds end-to-end if the provider is also configured for zero retention / opt-out of training.
Expected concurrency and per-user request volume?
Drives the rate-limiting and cost-guard design.
Accessibility / browser-support targets?
Affects reliance on third-party cookies for silent re-auth and on newer streaming APIs.
What a Strong Answer Covers
A strong answer treats the four constraints as forcing functions and stays internally consistent across every dimension below — never, for example, proposing localStorage for history while also claiming refresh resets.
Architecture & data flow:
a clean three-tier split (browser SPA ↔ stateless relay/BFF ↔ LLM provider) with the invariant that
all secrets stay server-side and all conversation content stays client-side
; the relay verifies the user JWT, holds the provider key, and re-streams tokens while storing nothing.
Streaming:
SSE-style chunked
fetch
+
ReadableStream
as the natural fit for a unidirectional token stream, with a crisp SSE-vs-WebSocket trade-off and a clear "use WebSockets only if you need duplex control" boundary. Bonus for correct frame buffering/parsing.
Session management:
conversation state in a JS in-memory array (not any persistent store), auth tokens in memory, and a coherent story for
why
refresh resets and how to avoid forcing a full re-login on every reload (same-origin silent re-auth / BFF).
Security:
OAuth 2.0 Authorization Code +
PKCE
(public client),
state
/
nonce
validation, full JWT verification (iss/aud/exp/signature via JWKS), provider key isolated to the relay, short-lived scoped tokens, and transport/injection hardening (HTTPS/HSTS, strict CORS allowlist, CSP, sanitized Markdown rendering, why XSS is the top threat when tokens live in memory).
Rate limiting:
layered, content-free defenses (edge/WAF → short-TTL token bucket keyed on the OAuth
sub
→ provider quotas → payload/
max_tokens
cost guards), with honesty about how a no-durable-store limiter is weaker than a shared store.
Failure & retry:
mid-stream drops, 429 with
Retry-After
+ exponential backoff with jitter, 5xx retries, timeouts via
AbortController
, token-expiry re-auth, and oversized-context handling — all anchored on the principle that
every recovery path re-sends context from the client
because there is no server-side resume point.
Trade-offs:
an honest ledger — privacy/compliance and operational simplicity vs. lost continuity, per-turn payload growth (full context re-sent every turn), weak observability, no personalization, and harder durable abuse control — plus mitigations that stay within the constraints (client-side summarization, user-driven export, content-free metrics).
Follow-up Questions
The conversation now exceeds the model's context window mid-chat. How do you handle it
client-side
(windowing, summarization, token estimation) without any server state, and what UX do you show the user?
The product adds
tool/function calling
that can interrupt or steer generation mid-stream. Does your transport choice change, and how do you keep the relay stateless?
A user reports "I clicked send twice and got billed twice." Walk through exactly what an idempotency/
request_id
key can and
cannot
guarantee in this no-durable-store design.
Support needs to debug a bad answer, but you store no conversation logs. What
content-free
observability can you add, and where do you draw the privacy line?