## Design an Evaluation Platform for a Design-Editing Agent A design application includes an agent that edits a user's document from natural-language instructions. A small request might be “change this button to red,” while a broad request such as “make this page more modern” has multiple acceptable results. Design an evaluation platform that can assess such changes and can also be adopted by engineers building new agents. ### Constraints & Assumptions - The platform must support both narrow deterministic edits and open-ended multi-step changes. - Users can accept or reject an agent's proposed change, but that signal is incomplete and potentially biased. - Offline evaluation needs versioned datasets and reproducible runs. - A new team should be able to integrate an agent and its evaluators without modifying the platform core. ### Clarifying Questions to Ask - Is evaluation performed before a change is shown, after it is shown, or both? - Can the platform inspect the document before and after the edit and the agent's action trace? - Which dimensions are hard constraints, and which require a graded score or human judgment? - How are privacy-sensitive design documents approved for offline use? ### Part 1 — Define Evaluation for Different Task Shapes Define the evaluation contract for a precise edit and for a broad visual request. Explain how task intent, acceptable outcomes, hard constraints, and graded quality dimensions are represented. #### What This Part Should Cover - Deterministic assertions for explicit properties such as the target element and color. - Rubrics or reference sets for open-ended quality rather than one exact expected document. - Preservation checks for parts of the design that should not change. - A decomposition of broad requests into observable dimensions. ```hint Separate correctness from preference A change can satisfy a requested property and still be rejected because it harms layout, consistency, or usability. ``` ### Part 2 — Design Online Measurement Use accept and reject interactions as one signal in an online evaluation system. Define the events, metrics, slices, and experiment controls needed to interpret that signal responsibly. #### What This Part Should Cover - Exposure, preview, accept, reject, undo, edit-after-accept, and abandonment events. - Denominators and task-level slices that prevent misleading aggregate rates. - Guardrails for latency, errors, repeated attempts, and destructive changes. - Recognition that user feedback is observational unless an experiment controls selection effects. ```hint Reconstruct the full funnel An acceptance rate is uninterpretable if failed generations and previews that users never opened are missing from the denominator. ``` ### Part 3 — Build an Offline Dataset and Evaluators Describe how examples are selected, labeled, versioned, and protected. Combine deterministic, model-based, and human evaluation where appropriate, and explain how evaluator quality is checked. #### What This Part Should Cover - Representative task strata from simple property edits through broad transformations. - Input document snapshots, instructions, allowed-change scopes, and expected evidence. - Agreement and calibration for subjective labels or rubric scores. - Frozen dataset and evaluator versions with regression and contamination controls. ```hint Treat the evaluator as a tested component An evaluator needs known positive, negative, and boundary examples before its score can gate an agent release. ``` ### Part 4 — Expose the Platform to Other Agent Teams Define what an engineer must provide to evaluate a new agent and what the platform returns. Cover execution isolation, extensibility, reproducibility, and comparison between agent versions. #### What This Part Should Cover - A versioned agent adapter, task dataset, evaluator configuration, and resource policy. - A common run and result schema with per-example evidence. - Queued execution, retries, cancellation, and immutable run manifests. - Side-by-side comparison with statistically and practically meaningful deltas. ```hint Make integrations declarative Prefer a registered manifest of agent, dataset, evaluators, and runtime limits over team-specific orchestration code. ``` ### What a Strong Answer Covers - Evaluation semantics that change appropriately with task ambiguity. - Online and offline evidence that complement rather than substitute for each other. - Reproducible, privacy-aware datasets and validated evaluators. - A platform contract that lets another team integrate without bypassing isolation or governance. ### Follow-up Questions 1. How would you evaluate “make this page more modern” without forcing one preferred visual style? 2. What does an immediate user undo reveal that a raw accept event does not? 3. How would you detect that a model-based evaluator favors the agent model being tested? 4. Which evidence would block a release even if the average quality score improved?