Experiment Design: Pin Unread Accounts at Top of Account Switcher

Context

You propose a feature for users who own multiple accounts (same person_id): when they open the account switcher, any accounts with UNREAD notifications are pinned to the top. The goal is to decide whether to launch this feature based on a rigorous experiment and decision framework.

Assume:

• The feature affects only multi-account users (≥2 accounts per person_id).
• A person can use multiple devices and platforms (web/iOS/Android).
• The account switcher lists all accounts associated with the same person_id.

Tasks

Design a concrete experiment and decision framework, then answer the scenario below. Be specific and justify trade-offs:

1. State the primary hypothesis and the target population. Define the unit of randomization and why (e.g., person_id-level cluster vs account-level), and how you will prevent cross-account contamination when a person has accounts in both variants.
2. Select one Overall Evaluation Criterion (OEC) and 3–5 secondary/diagnostic metrics. For each metric, define the exact numerator/denominator and attribution window (e.g., "unread resolution rate within 24h", "cross-account message send rate per person-day", "account-switch CTR", "median dwell time on switched-to account", "notification overload complaints per 1k users", etc.). Explain why the OEC is preferable to simpler clicks/CTR metrics.
3. List 3+ guardrails (e.g., retention, session length, error rate, notification latency, sample ratio mismatch) and the thresholds that would trigger a halt.
4. Specify the exposure and analysis windows (e.g., 28-day run with a 7-day cooldown), novelty/learning effects handling, and how you will handle users who gain/lose accounts mid-experiment.
5. Outline your power analysis inputs and outputs: baseline(s), minimal detectable effect (MDE), variance reduction methods (e.g., CUPED or covariate adjustment), and expected sample size/time to reach 80% power at alpha=0.05. You may assume realistic baselines but must show the formulas or the exact inputs you would plug into a calculator.
6. Data quality and attribution: define 'visit', 'dwell time', 'unread resolved', and how you will attribute outcomes when users switch across web/mobile (cross-device identity). Describe logging you need to add.
7. Decision scenario (interpretation test): Suppose the A/B test shows +40% lift in account-switch CTR to pinned unread accounts, but (a) median dwell time on the switched-to account is <5 seconds, (b) no statistically significant lift in reply/send-message rate or 'unread resolved within 24h', and (c) a +3% increase in bounce-back to the previous account within 10 seconds. Is this a successful outcome under your OEC? Give a yes/no decision and defend it. Propose at least two next steps (e.g., change ranking logic, add richer previews, segment by notification type or user intent) and how you would test them.