An AI product has been tested with a limited group of pilot users. The product team asks whether it should expand access. Describe how you would analyze the pilot, identify the metrics that matter, and present a recommendation despite selection bias and limited sample size. ### Constraints & Assumptions - Pilot participation was not fully randomized. - Usage logs and user attributes are available, but some important outcomes arrive with delay. - The pilot is small enough that averages may hide substantial user-level variation. - The recommendation may be to expand, run a stronger test, modify the product, or stop. ### Clarifying Questions to Ask - Who was eligible, who joined the pilot, and how were participants selected? - What user task is the AI product meant to improve, and what is the current alternative? - Which outcomes measure value, quality, safety, cost, and sustained adoption? - What logging exists for prompts, outputs, edits, failures, and downstream actions? - Which user groups or use cases are in scope for a possible expansion? ### What a Strong Answer Covers - A metric tree that separates adoption, task success, output quality, efficiency, retention, safety, and operating cost. - Cohort and funnel analysis at a clearly defined user, task, or session grain without treating repeated events as independent users. - Segmentation by use case and user characteristics, with uncertainty shown for small samples. - Recognition that pilot users are selected and that before-after improvement alone is not a causal estimate. - Qualitative review or calibrated human evaluation for output quality when logs cannot capture correctness. - A staged recommendation with explicit evidence gaps, experiment design, guardrails, and rollback criteria. ### Follow-up Questions 1. How would you measure value when users heavily edit an AI output before accepting it? 2. What would high initial usage followed by rapid decline suggest? 3. How would you compare pilot users with nonparticipants without overstating causality? 4. Which safety metric could justify stopping an expansion even if productivity improves?