Assess ranking change and design experiment

A multi-account product currently orders a user's accounts by **most recent visit**. The product team wants to change the ranking so that accounts with the **highest notification count** appear first, because they believe users are more likely to engage when the account with the most pending activity is surfaced at the top. If two accounts tie on notification count, break ties by most recent visit. Assume you have historical logs at the **user-session-account** level containing, for each session: - the ordered list of accounts shown - each account's notification count at session start - each account's last-visit timestamp - whether the user selected that account - whether the user opened any notification - downstream outcomes such as conversion/purchase and 7-day retention How would you: 1. Use historical data to assess whether this ranking change is likely to be beneficial before launching it? 2. Design an A/B test to measure the causal impact of the new ranking? Your answer should specify: - the eligible population - the unit of randomization - primary metric(s), secondary metrics, and guardrails - how you would handle confounding and selection bias in the historical analysis - what to do for sessions where the old and new rankings are identical - how to think about power, minimum detectable effect (MDE), and experiment duration - key segments and edge cases to inspect