Diagnose Low Adoption of an AI Contract-Review Agent and Define Its Success Metrics

A company that builds AI tools for lawyers has launched an AI agent that reviews contracts for enterprise legal teams, but adoption has come in below expectations. How would you diagnose why, define what success means for the agent, and decide what to do next? ### Clarifying Questions - What expectation is adoption falling short of: a forecast, a usage target agreed with customers, or the adoption of earlier features? - Who decides to use the agent: each lawyer individually, or does the customer roll it out to whole teams? - How long has the agent been available, and is the shortfall spread across customers or concentrated in a few? - Which contract types and workflows does the agent support, and where do lawyers reach it during their work? - What is logged: entry-point views, review runs, accepted, edited or rejected suggestions, exports? ### Part 1 — Diagnose the low adoption Explain how you would find where and why adoption falls short. ```hint Find the leak Adoption is a chain of steps from having access to coming back for more. Ask which step loses the most people, and for which customers. ``` ```hint Usage is not success A lawyer who runs the agent once and then rewrites its output has adopted it by some definitions. Ask what you would see in the data if the output were not good enough. ``` #### What This Part Should Cover - A step-by-step breakdown of adoption with a measurable definition for each step - Segmentation that localizes the problem, such as by customer, role, contract type and cohort - Hypotheses for each drop, and the data or user research that would confirm them ### Part 2 — Define success metrics Propose the metric you would use as the agent's North Star, plus supporting metrics and guardrails. ```hint Outcome over activity Ask what a legal team actually gets when the agent works, and whether a count of clicks or sessions captures it. ``` #### What This Part Should Cover - A North Star tied to completed, usable work rather than feature usage - Quality and customer-value metrics, with how each would be measured - Guardrails for the errors that matter most in legal work ### Part 3 — Decide what to do next Explain how you would choose among possible fixes and how you would know whether a fix worked. ```hint Few customers Enterprise customers are few and large. Ask what that means for running an experiment across them. ``` #### What This Part Should Cover - Mapping each diagnosis to a specific intervention, and prioritizing among them - An evaluation plan that works with a small number of large accounts - A decision rule, including when to change the product rather than its onboarding ### What a Strong Answer Covers - Clarifying what "adoption" and "expectations" mean before diagnosing - A complete funnel with clear definitions, and segmentation to find where it leaks - Output quality and customer value treated as part of adoption, not as separate topics - A North Star based on completed, usable outcomes, with guardrails for legal risk - Next steps that follow from the diagnosis, with a realistic way to measure their impact ### Follow-up Questions - Lawyers choose which contracts to run through the agent. How would you estimate the time it saves without selection bias? - After a change that makes the agent's output shorter, completed reviews rise but expert audits find more missed issues. How do you read that? - One large customer accounts for most of the usage. How would you report adoption so that it does not hide everyone else? - How would you set next quarter's target for the North Star?