Calibrate LLM output to match Word formatting
Overview
This question evaluates a candidate's ML system design skills for calibrating large language model outputs to strict document formatting schemas, covering schema representation, constraint enforcement, validation and repair, latency trade-offs, fallback strategies, and evaluation metrics.
Scenario
You’re building an LLM-powered feature in a word processor (e.g., Microsoft Word) that generates content users can insert directly into a document (headings, bullets, tables, citations, styles, etc.). A common failure mode is that the LLM’s output does not conform to the required Word formatting/spec (wrong heading levels, broken lists, invalid table structure, missing citations, inconsistent styles).
Task
Design an approach to calibrate and enforce that the LLM’s generated content matches a target Word formatting specification.
Requirements
- Output must be valid according to a predefined schema (e.g., Word OpenXML subset or an internal document model).
- Low latency for interactive generation.
- Minimize “format drift” across revisions and multi-turn edits.
- Provide a safe fallback when the model cannot comply.
What to cover
- What representation/schema you generate (e.g., structured JSON AST, XML, markdown-like intermediate form).
- How you enforce constraints at generation time vs post-processing.
- Training/fine-tuning or preference optimization options.
- Validation, automatic repair, and human-in-the-loop strategies.
- Metrics and offline/online evaluation.