This question evaluates a candidate's competency in ML system design and backend engineering for conversational AI, including architecture, session state and context management, streaming responses, resilience (rate limiting and error handling), observability, lightweight persistence, and safety when integrating a large language model API.
Design and implement a minimal yet functional conversational chatbot using the OpenAI API within a two-hour coding exercise. Specify the architecture (API server, message schema, session state, context window strategy), how you will stream responses, and how you will handle rate limits, retries, and errors. Describe logging/metrics for quick debugging, a lightweight persistence option, and basic safety checks. Outline your project structure, key endpoints, and exact steps to run locally; you may optionally leverage an AI IDE like Cursor—explain how it speeds up delivery.
Quick Answer: This question evaluates a candidate's competency in ML system design and backend engineering for conversational AI, including architecture, session state and context management, streaming responses, resilience (rate limiting and error handling), observability, lightweight persistence, and safety when integrating a large language model API.
mediumSoftware EngineerTechnical ScreenML System Design
28
0
Design and Build a Minimal Conversational Chatbot (2-Hour Coding Exercise)
You are given an OpenAI API key and two hours. Design and implement a minimal but genuinely functional conversational chatbot that runs locally. This is a hands-on, time-boxed build: the interviewer cares about a working vertical slice, sound engineering judgment under a clock, and clean trade-off narration — not an exhaustive feature checklist.
Your chatbot must let a client start a session, send messages, and receive replies, while addressing the following dimensions:
Architecture
— an API server, a message schema, session state, and a context-window strategy (how you fit a growing conversation into the model's token limit).
Streaming
— stream the assistant's reply to the client token-by-token.
Operations
— logging and metrics for fast debugging, a lightweight persistence option, and basic safety checks (content moderation / guardrails).
Deliverables
— your project structure, the key API endpoints, the exact steps to run locally, and (optionally) how an AI IDE such as Cursor speeds up delivery.
You may use any mainstream language/framework (e.g. Node.js + Express, or Python + FastAPI). State your choice and justify it briefly.
Constraints & Assumptions
Time box:
~2 hours, solo, local machine. Prioritize a working slice over completeness.
Scope:
single-process service is fine; no multi-tenant auth, horizontal scaling, or production hardening required.
Persistence:
in-memory is acceptable for the demo, but show the swap to a durable store (e.g. SQLite) as a clear, small change.
Resources:
a valid OpenAI API key with rate limits that can trip under load; assume the model has a finite context window.
Safety:
assume some user inputs may be unsafe and must be screened.
Clarifying Questions to Ask Guidance
What does "functional" mean for grading — a CLI/HTTP demo that returns a reply, or a polished UI? (Sets how much time goes to the client.)
Single user / single session, or must sessions be isolated and concurrent?
Is durable persistence required, or is in-memory acceptable with a documented upgrade path?
Which model and roughly what context limit should I target, and is cost a constraint?
Are streaming responses mandatory, or is a non-streaming JSON reply acceptable as the baseline?
What level of safety/moderation is expected — input screening only, or output screening too?
What a Strong Answer Covers Guidance
Time-boxed sequencing:
ships a working vertical slice first, then layers streaming → context → resilience → ops, each independently demoable; narrates trade-offs ("in-memory now, one-line swap to SQLite").
Correct API usage:
calls the OpenAI API correctly (chat generation and moderation) and handles the streaming event types accurately
without inventing API surface
.
Architecture & schema:
a thin API layer, a flat serializable message/session schema, and a session store behind a DAO so the persistence backend is swappable.
Token-budgeted context strategy:
reserves output tokens, keeps a rolling summary, packs recent turns within budget, and summarizes-on-overflow; uses a single source of truth for the current user turn.
Streaming design:
SSE end-to-end, with a clear stance on retry behavior once streaming has begun.
Resilience:
transient-only retries with backoff + jitter,
Retry-After
honored, a uniform error envelope, and sensible status-code mapping.
Operations:
structured logging with a correlation id (no raw PII), useful metrics, a
/health
and
/metrics
endpoint, and input (and ideally output) moderation.
Deliverables & runability:
a coherent project layout, named endpoints, and exact local run/smoke-test commands.
Follow-up Questions Guidance
Your
chars / 4
token estimate diverges for code and non-Latin scripts — when does that bite, and how would you swap in a real tokenizer without slowing the build?
A retry fires
after
the first streamed delta has reached the client. Walk through exactly what goes wrong and how your design prevents duplicated output.
How would you make sessions survive a process restart and scale to multiple server instances? What changes in the session store and the context packer?
The input moderation check passes but the model still generates something unsafe. How do you catch it, and what's the latency cost of moderating the output stream?
If you used Cursor (or any AI IDE) to scaffold the OpenAI streaming code, how do you guard against it hallucinating event names or endpoints?