Facebook login security and a 30% satisfaction gap: diagnose causes, critique the survey

Facebook surveyed two groups of users, 50,000 in each, about how satisfied they are. Users in the first group, labeled the experiment group, had certain login security features turned on; users in the second group did not. Satisfaction in the first group came out 30% lower than in the second. Explain what could be behind the gap, and evaluate how the survey was run. ### Clarifying Questions - Were users randomly assigned to have the features turned on, or did each user decide whether to enable them? - Which features are these, for example a second verification step at login or alerts about logins from new devices, and how often does each one interrupt a user? - Does "30% lower" compare average scores, or the share of respondents who said they were satisfied? - What did the survey ask, and when and through which channel did each group receive it? ### Part 1 — Explaining the gap List the most plausible explanations for why the first group reports lower satisfaction, rank them, and say what data you would look at, in what order, to confirm or rule out each one. ```hint Compare more than the features Ask how each user came to be in their group, and whether the two groups could differ in ways that have nothing to do with the login flow. ``` #### What This Part Should Cover - Explanations grouped into a real effect of the features, differences in who is in each group, and measurement artifacts - A concrete mechanism for each explanation, tied to how people log in and use their accounts - An order of investigation and the data that would confirm or rule out each explanation ### Part 2 — Critiquing the survey Assess the survey as a way to measure how the features affect satisfaction: how the groups were formed, who was sampled and who answered, how the questions were worded and timed, and how the 30% figure was computed. Then describe how you would redesign the study so that a gap like this could be attributed to the features. ```hint Size is not the same as validity Think about what a 50,000-person group protects against and what it does not, and about every way the survey experience itself might have differed between the groups. ``` #### What This Part Should Cover - Whether the comparison can attribute the gap to the features, and what a valid comparison would need - Sampling, response, wording and timing problems that could create or inflate the gap - A redesigned study and the outcomes it would measure besides stated satisfaction ### What a Strong Answer Covers - An assessment of whether the two groups were comparable before the survey, and how to test it - Friction mechanisms specific to login security, weighed against the protection the features provide - A view on statistical versus practical significance given the group sizes - A decision rule for keeping, changing or rolling back the features once the cause is known ### Follow-up Questions - If a randomized test confirmed a smaller but real drop in satisfaction alongside fewer hijacked accounts, how would you decide whether to keep the features? - Which behavioral metrics would you track next to the survey to see whether the features are hurting usage? - How would you reduce the friction for the users who most need the protection without weakening it?