Signup test lifts profile completion but cuts 7-day retention: diagnose and decide

Your team launched a new signup flow designed to get new users to fill in more of their profile. The A/B test shows that the share of users who added more profile information rose by 8%, while 7-day retention fell by 2%. What do you do? ```hint Check who each number counts Before weighing the two results against each other, pin down which users sit in the denominator of each metric and at which step of the new flow a user could drop out. ``` ### Clarifying Questions - Are the 8% and 2% relative changes or percentage-point changes, and is each one statistically significant? - What kind of product is this, and what is the extra profile information used for once it is collected? - Is the new profile step required or skippable, and where does it sit in the signup flow? - Is 7-day retention measured over every user assigned to the test, or only over those who finished signing up? ### What a Strong Answer Covers - Checks that the experiment's result is trustworthy before interpreting it - Which outcome the business ultimately cares about, and why the profile information was wanted in the first place - Concrete hypotheses for how one flow could raise profile completion and lower retention, each paired with the data cut that would confirm or rule it out - A decision (ship, iterate or roll back) with explicit criteria, plus the next experiment and its primary and guardrail metrics ### Follow-up Questions - If 30-day retention came out equal in both arms, would your decision change? - How would you weigh one extra completed profile against one lost retained user? - The team that requested the profile data says a new recommendation model needs it now. How do you respond?