Batch LLM Document Enrichment Pipeline With Fleet-Wide 429 Backoff, Dedup, and Replay

A document platform wants to enrich its stored documents with metadata produced by an external large language model (LLM) API. A batch process sends documents to the third-party API and writes each result back to the platform. Design this enrichment pipeline, then handle the follow-ups the interviewer focuses on: rate limiting by the provider, deduplication of work across workers, replaying recent data, and observability. ### Constraints and Clarifications - The LLM API is owned by a third party. When the caller exceeds its limits, it responds with HTTP 429 and may include a `Retry-After` header. - Work is spread across a fleet of workers that run concurrently. - Document counts, the provider's quota, per-call latency, and freshness targets were not specified. Ask for them, or state explicit assumptions before sizing anything. - Assume a document can change after it has been enriched, so a result must be tied to the document content it was computed from. ### Clarifying Questions - Is the provider's quota enforced per API key or account (shared by every worker), or per worker or connection? - Is enrichment safe to repeat for the same document content, and is the output expected to be stable enough that a re-run can simply overwrite the previous result? - Where do results live: on the document record itself, or in a separate enrichment store keyed by document? - How quickly must a new or changed document be enriched, and is it acceptable for part of the corpus to remain unenriched during a provider outage? ### Part 1 — Core Batch Pipeline Describe how documents are selected for enrichment, how work is distributed to workers, how the external API is called, and how results are written back. ```hint Track work explicitly Think about what durable record tells you, for each document, whether it still needs enrichment, is being worked on, has finished, or has failed. ``` #### What This Part Should Cover - How documents needing enrichment are discovered and recorded. - How work is distributed across the fleet and how results are persisted against the correct document content. - How transient failures differ from permanently bad inputs. ### Part 2 — The Provider Starts Returning 429 The workers start receiving HTTP 429 responses from the third-party API. What should happen, and why is it not enough for each worker to handle the 429 on its own? ```hint Who owns the quota Consider what the 429 response tells you, and whether the limit it reports applies to one worker or to all of them together. ``` #### What This Part Should Cover - Why independent per-worker behavior fails when the quota is shared. - How the whole fleet learns about and respects the provider's back-off signal. - How the fleet resumes without immediately triggering another wave of 429 responses. ### Part 3 — Deduplicating Work How do you guarantee that two workers do not process the same document at the same time, and that a worker crash does not leave a document stuck forever? ```hint Claims that can expire Consider what makes taking ownership of a document atomic, and what should happen to that ownership when its owner stops responding. ``` #### What This Part Should Cover - An atomic way for a worker to claim documents. - Recovery of claims held by crashed or stalled workers. - Why a late or duplicate completion must not corrupt stored results. ### Part 4 — Replaying the Past 5 Days Suppose you need to reprocess everything from the past 5 days, for example after fixing a bug in the enrichment output. How does the design support this? ```hint Select by time, reuse the machinery Think about which recorded timestamp defines the 5-day window, and how replayed work can flow through the same claiming and rate-limiting path as normal work. ``` #### What This Part Should Cover - How the replay window is selected and resumed if the replay is interrupted. - How replay shares the provider quota with fresh work. - Why replayed writes are safe to apply over existing results. ### Part 5 — Observability What metrics, logs, and alerts would you add so that the team knows the pipeline is healthy and can diagnose it when it is not? ```hint Symptoms before causes Start from what a user of the enriched data would notice first, then add signals that explain why it happened. ``` #### What This Part Should Cover - Backlog, throughput, and end-to-end lag signals. - Error signals broken down by class, including rate limiting and permanently failed documents. - Alerts that are actionable rather than noisy. ### What a Strong Answer Covers - A durable per-document work state that drives selection, claiming, retry, and replay. - Rate-limit handling coordinated across the entire worker fleet, not just inside one process. - Idempotent, version-aware result writes so retries, duplicates, and replays are harmless. - Clear separation of transient, rate-limit, and permanent failures, with a place for failed work to go. - Metrics and alerts that surface backlog growth, provider throttling, and stuck work. ### Follow-up Questions 1. The shared cache that holds the fleet's back-off state becomes unavailable. How should workers behave? 2. A replay of 5 days would take longer than the provider's daily quota allows. How do you prioritize it against fresh documents? 3. Some documents exceed the model's input limit. How does the pipeline handle them? 4. How would you detect that the provider has started returning well-formed but empty or low-quality enrichment results?