Evaluate Notification-Based Account Ranking

A product allows users to switch among multiple accounts. Today, the account switcher ranks accounts by **most recent visit**. The product team wants to change the ranking so that the account with the **highest number of unread notifications** appears first, under the hypothesis that users should be taken more quickly to the account that needs attention. How would you evaluate whether this ranking change is useful? Please answer in two parts: 1. **Historical-data analysis (before launch):** Using existing logs, how would you assess whether ranking by unread notifications is likely to outperform ranking by recency? 2. **A/B test design (after launch):** How would you set up a randomized experiment to measure the causal impact of this change? In your answer, be explicit about: - The product objective and how it affects metric choice - Primary metrics, secondary metrics, and guardrail metrics - How you would define the analysis population and experiment unit - Biases and confounding in the historical analysis - Sample size, MDE, experiment duration, and variance reduction ideas - Important edge cases such as users with only one account, ties in ranking, very high-notification accounts, and novelty effects