Find the Bugs in an AI-Written A/B Test Analysis, Then Rewrite It in Plain Python

In a data scientist technical screen, you are given a short Python script, written by an AI assistant, that analyzes a basic A/B test, and asked to find what is wrong with it. You then write the same analysis yourself in Python. The online editor offers an AI assistant, but no third-party packages are installed, so pandas, NumPy and SciPy are not available. The report does not reproduce the script or describe the experiment. This version supplies a stand-in of the same kind: a test on a conversion rate, analyzed by a script that contains several mistakes. The script receives `rows`, a list of dictionaries from the experiment's exposure log. Each row is one visit by a user who was assigned to the experiment: | Key | Meaning | | --- | --- | | `user_id` | User identifier; a user can have several rows, one per visit | | `group` | `"control"` or `"treatment"`, assigned per user | | `converted` | `1` if the user made a purchase during that visit, otherwise `0` | The metric is the user-level conversion rate: the share of assigned users who made at least one purchase. The planned split is 50/50, and the test is two-sided at the 5% significance level. ```python import math def analyze_ab_test(rows): n = {"control": 0, "treatment": 0} conversions = {"control": 0, "treatment": 0} for row in rows: n[row["group"]] += 1 conversions[row["group"]] += row["converted"] p_c = conversions["control"] / n["control"] p_t = conversions["treatment"] / n["treatment"] lift = (p_t - p_c) / p_t p_pool = (conversions["control"] + conversions["treatment"]) / (n["control"] + n["treatment"]) se = math.sqrt(p_pool * (1 - p_pool) * (1 / n["control"] + 1 / n["treatment"])) z = (p_t - p_c) / se cdf = 0.5 * (1 + math.erf(z / 2)) p_value = 2 * (1 - cdf) ci_low = (p_t - p_c) - 1.645 * se ci_high = (p_t - p_c) + 1.645 * se return { "control_rate": p_c, "treatment_rate": p_t, "relative_lift": lift, "z": z, "p_value": p_value, "ci_95": (ci_low, ci_high), "significant": p_value < 0.05, } ``` ### Clarifying Questions - How should a user who appears in both groups be handled? - Should the analysis verify that the observed split matches the planned 50/50 split? - Should the confidence interval be for the absolute difference in conversion rates, the relative lift, or both? - Can a group be empty, or can every user in the test have the same outcome? ### Part 1 — Review the script Find every problem in the script. For each one, explain its effect on the reported numbers or on the decision, and give the fix. Also say which parts are correct. ```hint Check the unit Compare the unit the metric is defined on with the unit the loop counts. ``` ```hint Test the formulas with known values Plug in a z statistic whose two-sided p-value you know, such as 1.96, and then try a negative one. ``` #### What This Part Should Cover - The unit-of-analysis mistake and its effect on the sample size and the standard error - Errors in the formulas: the lift, the normal CDF, the two-sided p-value and the interval - Missing safeguards: invalid labels, empty groups, users in both groups, and the split check ### Part 2 — Write the analysis yourself Write `analyze_ab_test(rows, alpha=0.05)` using only the Python standard library. Return, for each group, the number of users, the number who converted and the conversion rate; the absolute difference and the relative lift; the z statistic and two-sided p-value of a two-proportion z-test; and a confidence interval for the difference at level 1 − alpha. Include the checks you consider necessary before trusting the result. ```hint No SciPy, still a normal distribution You need the normal CDF and its inverse. Check what the standard library already provides before writing either yourself. ``` #### What This Part Should Cover - Aggregation to one record per user, with a stated rule for users seen in both groups - A correct two-proportion z-test, two-sided p-value and interval built from the standard library only - Validation of the inputs and of the split, and a way to check the code without SciPy ### What a Strong Answer Covers - Reading AI-written code critically: checking units, formulas and edge cases instead of trusting plausible output - Correct statistics for a difference in proportions, with the appropriate standard error for the test and for the interval - Clean, testable code that runs with no third-party packages - Sanity checks before interpretation: assignment integrity, the sample ratio, and degenerate inputs - A clear statement of the result and the decision it supports ### Follow-up Questions - How would you check your implementation's p-values without SciPy? - If the metric were revenue per user instead of conversion, what would change in your code? - If the metric were purchases per visit rather than per user, why would a two-proportion test on visits be wrong, and what would you do instead?