Design a Backend That Answers One Question Across 100,000 Documents with an LLM

Design a backend system that uses a large language model (LLM) to answer the same user-provided question across a large set of documents. The user supplies one question and a set of documents; the system asks the question of each document and returns a table in which each row corresponds to one document, with one column for the document name and another for the answer drawn from that document. The system must handle on the order of 100,000 LLM requests. The report gives only this statement, so document sizes, the LLM provider's limits, how soon results are needed and cost targets are open: settle them with the interviewer before designing. ```hint Find the real bottleneck Estimate the tokens a job sends, then ask which limit a 100,000-request job reaches first: your own servers, or the quotas of the LLM provider you call. ``` ```hint One row, one definite outcome Ask what each row of the table shows while the job runs, when a document does not contain the answer, and when its request fails, and how a crashed worker's rows get finished without paying for them twice. ``` ```hint Documents that do not fit Ask what happens when a document is longer than the model's context window, and how that changes both the number of requests and the way one answer per document is formed. ``` ### Clarifying Questions - Is 100,000 the number of documents in one job, the number of LLM calls in one job, or the total load across users, and how many jobs run at the same time? - How large are the documents and in what formats (text PDFs, scans that need OCR, office files)? Can one be longer than the model's context window? - Must the whole table be ready within minutes, or is a job that finishes in hours acceptable, and should rows appear as they complete? - Which LLM provider or providers are used, and what request, token and cost limits apply? - Must each answer point to the passage it came from, and what should a row show when the document does not contain the answer? - What confidentiality and isolation rules apply to the documents and the answers? ### What a Strong Answer Covers - Requirements and estimates in tokens, requests, duration and cost, showing which limit dominates - An asynchronous job model with a durable record per document row and incremental results - Dispatch against provider rate limits: admission by tokens, backoff on overload, fairness across jobs and tenants - Handling documents longer than the context window while still producing one answer per document - Idempotent retries, partial failures, cancellation and provider outages - Answer quality, cost control, observability and data isolation ### Follow-up Questions - The provider allows a fixed number of tokens per minute, and one customer submits a 100,000-document job while others submit small jobs. How do you schedule them? - The user edits the question slightly and reruns it on the same documents. What can you reuse, and what must be recomputed? - How would you evaluate the answers and detect an answer that does not come from the document? - Halfway through a job the provider is unavailable for 20 minutes. What happens to the job, and what does the user see?