Design and justify unread-account pinning experiment

You plan to launch a UI change for people who own multiple accounts: pin accounts with unread notifications to the top of the account switcher and sort those unread accounts first. Design a rigorous experiment and decision framework. Provide: 1) Experiment design - Target population and inclusion/exclusion criteria. - Randomization unit and clustering (e.g., person_id-level so all of a person’s accounts get the same treatment) to avoid cross-account contamination; describe assignment, exposure, and bucketing logic. - Duration, ramp plan, and any holdouts. 2) Metrics and success criteria - Choose ONE primary success metric that directly captures value from pinning. Define it precisely at the person-day level (e.g., unread-to-read conversion within 24 hours across all owned accounts) and justify why it is better than generic engagement metrics. - List key secondary metrics and guardrails (e.g., time to clear first unread, subsequent actions such as reply/send, N-day retention, support tickets, block/report rate). Define directional expectations and acceptable deltas. - Specify the decision rule for ship/no-ship (e.g., minimum detectable effect, confidence level, correction for multiple metrics), and discuss how you’ll handle novelty effects and learning/adaptation. 3) Analysis plan - Detail how you’ll attribute outcomes when a person owns N accounts but only some have unread; how repeated exposures are handled; and how you’ll prevent metric inflation from simply increasing switches without clearing unreads. - Pre-registration of slicing: by number of accounts (2, 3, >3), baseline engagement, and device type. - Bias controls: CUPED or covariate adjustment, and how you’ll diagnose interference (e.g., control users messaging treated users). 4) Instrumentation requirements - Exact events to log: impression of pinned list, account order, click-through to account, notification read state transitions, dwell time, and subsequent meaningful actions. - Data quality checks and invariants you will monitor (traffic balance, event firing rates). 5) Interpretation scenario Suppose during the test you observe: a large increase in clicks into the top pinned unread account among treated users, but median dwell time on that account page is only a few seconds and overall session length is flat. What additional diagnostics would you run and what thresholded evidence would convince you the feature is (a) a success, (b) neutral, or (c) harmful? Be specific about which metrics must move (e.g., person-level unread clearance rate within 24h, fraction of sessions with zero unreads at exit, downstream send/reply rate) and how you would weigh them against the short dwell-time signal.