Validation and Retry Design for Schema Drift in a Multi-Step LLM JSON Pipeline

You run a multi-step LLM extraction pipeline in which each step consumes the structured JSON produced by the step before it: step N depends on the JSON output of step N-1. Occasionally the upstream model returns output that deviates slightly from the expected schema, and the deviation causes failures further downstream. Design a validation and retry mechanism that keeps token waste and processing latency low and never runs past the pipeline's strict timeouts. The round was framed as a practical architecture discussion rather than an algorithm problem. The report does not give the number of steps, the document sizes, the timeout values or the models used, so settle these with the interviewer. ```hint Where the failure surfaces Ask at which point a malformed output is cheapest to catch, and what it costs when the failure only shows up in a later step. ``` ```hint What a retry needs to see Think about how much of the original input the model actually needs in order to fix an output that is almost right. ``` ```hint Budget, not count Consider deciding whether to retry from the time left before the deadline rather than from a fixed number of attempts. ``` ### Clarifying Questions - How many steps are there, and how large are each step's inputs and outputs (tokens of context versus tokens of JSON)? - Is the timeout an end-to-end deadline per request, a limit per step, or both, and is a partial result acceptable when it is reached? - What do the deviations look like in practice: invalid JSON, extra text around it, missing or misnamed fields, wrong types, truncated output, or well-formed values that are wrong? - Does the model or provider offer a structured-output or constrained-decoding mode, and at what temperature do the steps run? - Does the pipeline serve interactive requests or run as batch jobs, and which steps are deterministic code rather than model calls? ### What a Strong Answer Covers - An explicit, versioned contract at every step boundary, with strict runtime validation where each output is produced - Classification of failures by type, with a remedy matched to each type and ordered by cost - A low-token correction path, and a clear rule for when a fuller re-run is unavoidable - Deadline propagation and budget-aware retry decisions, with caps, fallbacks and defined behaviour when time runs out - Caching and checkpointing of intermediate results so that retries resume at the failing step - Observability of failure types, repair success, retry tokens and latency, plus protection against retry storms ### Follow-up Questions - The correction prompt returns valid JSON, but it filled in a required field it had no way of knowing from the invalid output alone. How do you detect and prevent that? - The schema of step N's output has to change. How do you roll the change out without breaking in-flight requests or the steps after it? - The model degrades and most outputs fail validation for several minutes. What keeps the retry mechanism from making the situation worse? - How would you test the mechanism before relying on it in production?