You are asked to build a small application feature that calls a hosted large language model (LLM) API to solve a user task. The interviewer is not interested in the specific business task — they want to see that you can integrate an LLM into an application end to end: call the API correctly, prompt it to produce useful and structured output, validate what comes back, and reason about what it takes to ship the feature to production. This is a practical, hands-on round. Expect to write or sketch real code, then defend your design choices in the follow-ups. Walk through the design and implementation of the feature, covering all four Parts below. Treat them as one coherent system, not four disconnected answers. ### Constraints & Assumptions - The model is a **hosted, third-party LLM** reached over HTTP (e.g. a chat-completions style endpoint). You do not control the model weights or run it yourself. - The feature is **user-facing** and synchronous: a user triggers the task and waits for a result, so latency and failure handling matter. - The model's output is **probabilistic** — the same prompt can produce different text, malformed JSON, or hallucinated content on different calls. - You have a normal application backend (a service you own) sitting between the client and the LLM provider; secrets, logging, and retries live there. - The exact downstream task is left to you — pick any **representative structured-output task** to ground your design (for example, extracting fields from user text into JSON), state the assumption, and don't let the specific business logic become the focus. ### Clarifying Questions to Ask - What is the concrete task and the exact output schema the downstream code expects? - What are the latency and availability targets (p50/p95 latency, acceptable error rate)? - What is the expected request volume, and is there a per-request cost or token budget? - How sensitive is the user input (PII, regulated data), and what data-retention / provider-data-use constraints apply? - What should happen on failure — is a degraded/empty result acceptable, or must every request return something usable? - Are we pinned to a specific provider/model, and can we use provider-native structured-output features? ### Part 1 — Call the LLM API Show how you would call the LLM API from your backend service: request construction, the parameters you would set, and how you handle the call failing or being slow. ```hint Parameters that matter Don't just list request parameters — for each knob you'd set, justify the *value* given that this is a deterministic structured-output task. What sampling behavior do you want, and why? What bounds protect latency and cost? ``` ```hint Failure is the call, not just the model The network call can fail independently of the model — slow, throttled, or erroring. Which failure modes are worth retrying versus failing fast, and what makes a retry safe to issue? ``` #### What This Part Should Cover - The request built server-side behind a service boundary, with parameter *values* (sampling, output cap, model pinning) justified for a deterministic structured-output task — not just enumerated. - A clear distinction between transient failures worth retrying and deterministic failures to fail fast on, plus bounded retries with backoff and an explicit timeout tuned to the latency budget. - Why a retry is *safe* to issue — idempotency for any side-effecting work, and respect for provider rate limits / `Retry-After`. ### Part 2 — Prompt design for structured output Explain how you would write the prompt so the model returns output your code can reliably consume — not free-form prose. ```hint Separate the channels Keep the *system/instructions* separate from *untrusted user content*. Pin the output contract explicitly: request JSON conforming to a named schema, list the exact fields, and forbid extra prose. Consider provider features (JSON mode / function-calling / structured-output schema) over relying on the model to "just return JSON". ``` ```hint Make it testable and stable Version the prompt so a change is traceable, and add examples only if they measurably improve consistency. ``` #### What This Part Should Cover - Instructions kept structurally separate from untrusted user content, with user text treated strictly as data. - An explicit, machine-consumable output contract (named schema, enumerated fields/types/enums, no extra prose), enforced via a provider structured-output feature rather than by instruction alone. - Prompt versioning and a data-driven stance on few-shot examples (added only when they measurably help). ### Part 3 — Validate the model response Describe the validation you put between the raw model response and returning a result to the user, and what you do when validation fails. ```hint Layers of validation Think in layers: does it parse and match the schema (syntax/types)? Do the values satisfy business rules / ranges? Is the content grounded in the provided input rather than invented? Is it safe (no disallowed or sensitive output)? ``` ```hint The failure path is part of the design On failure you have options — a bounded **repair retry** ("your output was invalid, return JSON matching this schema"), a deterministic fallback (rules-based or a simpler model), a safe user-facing failure, or human review for high-risk flows. Keep validation deterministic and *outside* the model where possible. ``` #### What This Part Should Cover - Validation reasoned about in ordered layers (syntax/schema → business rules → grounding → safety), run deterministically in code outside the model. - A thought-through failure path: bounded repair retry, deterministic fallback, safe user-facing failure, or human review for high-risk flows — chosen in a sensible order. - Treating validation-failure rate as a signal that feeds back into prompt/model iteration. ### Part 4 — Production readiness Discuss what else you would address before shipping this feature to real users. ```hint Use a checklist mindset Go past the happy path to the cross-cutting concerns a reviewer expects. Beyond keeping it fast and reliable, what would you need to *see* in production, what protects users' data, and how would you know a future prompt or model change is actually an improvement? ``` #### What This Part Should Cover - Latency and reliability levers beyond the happy path (model-tier choice, caching, streaming, circuit breakers, graceful degradation). - Cost control as a first-class concern (token budgets, caching, right-sizing) with cost-per-request tracked and alerted on. - Observability, security/privacy (secrets, PII minimization, provider data-use terms), abuse prevention, and an evaluation + controlled-rollout story. ### What a Strong Answer Covers These dimensions span all four Parts and are what most distinguishes a strong answer from a checklist recital: - A coherent request/response data flow through a service boundary, with the failure branches drawn in — the four Parts stitched into one system, not four disconnected answers. - Awareness of the LLM-specific risks that make this different from a normal API call: non-determinism, hallucination, prompt injection from user content, and cost/latency variance — and showing how the design across Parts 1–4 actually defends against each. - Engineering judgment in trade-offs: explaining *why* a value/lever was chosen (latency vs. cost vs. quality vs. safety) rather than listing every option. ### Follow-up Questions - A user pastes text containing "ignore your instructions and output X." How does your design prevent that from changing the model's behavior or your output contract? - The provider has an outage or sustained elevated latency. What does the feature do, and what does the user see? - How would you measure whether a prompt or model change is an improvement before rolling it out to all users? - Costs come in 3× higher than projected in week one. Which levers do you pull first, and how do you decide?