Debug a Numerical Underflow in an A/B Testing Analytics Repo When the AI Agent Is Wrong

This round is an AI-assisted coding exercise. You are given a real, multi-file repository for an A/B testing analytics platform, a list of five reported issues, an AI coding agent you may use freely, and about 45 minutes. Fixing all five is not expected. The interviewer is watching how you work, and above all how you reason when the agent's suggestions are wrong. The actual repository is not reproduced here. For practice, assume a Python codebase that ingests per-user conversion events and computes experiment results, and that the hardest of the five issues is a numerical underflow bug. To make that issue concrete, assume it sits in a sequential test that decides whether an experiment can stop early: ```python import numpy as np def likelihood(outcomes: np.ndarray, p: float) -> np.float64: # outcomes holds one 0/1 conversion flag per user return np.prod(np.where(outcomes == 1, p, 1.0 - p)) def sequential_decision(outcomes, p0, p1, alpha=0.05, beta=0.2): ratio = likelihood(outcomes, p1) / likelihood(outcomes, p0) upper = (1 - beta) / alpha lower = beta / (1 - alpha) if ratio >= upper: return "stop_effect_detected" if ratio <= lower: return "stop_no_effect" return "continue" ``` The bug report says that small test fixtures behave correctly, but on real experiments with thousands of users the function never returns anything except "continue", and the logs show a runtime warning about an invalid value in a division. Each time you ask the agent to fix it, it hands back a variant that still fails. ### Constraints and Clarifications - About 45 minutes for five issues; partial completion is normal, and the order you choose is part of the evaluation. - You may use the agent for reading, searching, explaining and writing code, but you are accountable for every change you keep. - Think aloud: the interviewer needs to hear why you accept or reject what the agent produces. - The code above is a practice stand-in for the underflow issue; the real repository's version may differ in detail. ### Clarifying Questions - Are the five issues ranked by severity, and is there a test suite or a reproduction for each? - May I add or change tests, and must the public function signatures stay the same? - What behavior is expected at the boundaries: no users yet, every user converting, or no user converting? - Is writing a fix by hand acceptable, or should every change go through the agent? ### Part 1 — Plan the 45 minutes Before touching code, explain how you would get oriented in an unfamiliar multi-file repository and decide the order in which to attack five issues with limited time. Say how you would use the agent during this phase and how you would know that each fix actually worked. ```hint Reproduce before you repair Decide what evidence you want for each issue before any change is made, and which issues give the most value per minute. ``` #### What This Part Should Cover - Fast orientation: entry points, data flow, existing tests and how to run them - Triage by severity, confidence and cost, with a timebox per issue - A failing reproduction or test for each issue before it is fixed - Using the agent for exploration and summaries as well as for writing code ### Part 2 — Diagnose and fix the underflow Using the code above, explain exactly why it produces the wrong decision on large experiments, then write a fix that gives the same decisions on small inputs and correct ones at any realistic size. Add the test that would have caught the bug. ```hint Look at the magnitudes Estimate how small a product of thousands of numbers below one becomes, and compare it with the smallest positive double. ``` #### What This Part Should Cover - The precise chain of events from the product to the stuck decision - A fix that changes the numeric representation instead of patching the symptom - Careful handling of terms close to zero or one - A regression test at realistic scale plus boundary cases ### Part 3 — When the agent keeps getting it wrong Suppose the agent's fixes keep failing: for example, it adds a tiny constant to both likelihoods, switches to a wider floating-point type, or rescales the product, and the bug comes back. Explain how you recognize that a suggestion is wrong, how you steer the agent toward a correct fix, and when you stop prompting and write the fix yourself. ```hint Make the failure undeniable Think about what artifact would let you, the agent and the interviewer all see within seconds whether a proposed fix is correct. ``` #### What This Part Should Cover - Judging agent output against an independent check rather than by how plausible it looks - Giving the agent the root cause and constraints instead of repeating the same request - Recognizing when a manual fix is faster - Keeping the interviewer informed of the reasoning throughout ### What a Strong Answer Covers - Spends the first minutes on orientation, reproduction and triage rather than prompting blindly - Explains the root cause numerically, not just as "floating-point imprecision" - Fixes the computation in log space and proves it with tests at realistic scale - Treats the agent as a fallible collaborator whose output is reviewed like a pull request - Manages the clock openly: what was finished, what was left, and why ### Follow-up Questions - The same codebase also computes a Bayesian probability that one arm beats the other using Beta functions. Where could underflow or overflow appear there, and how would you guard against it? - How would you make sure this class of bug cannot reappear in other statistical functions in the repository? - With ten minutes left and two issues unresolved, how do you decide what to do, and what do you tell the interviewer? - How would you review a large multi-file change that the agent produced for one of the other issues?