### Prompt A social travel app wants to add a notification: **“Someone in one of your groups is live now.”** The notification can increase attendance at live sessions, but it may also create notification fatigue, shift users away from other valuable activity, and change the experience of group members who were not directly notified. Design an evaluation that determines the notification's effect on the overall product, not just notification clicks. Include a randomized design that addresses network effects and a retrospective alternative for a situation in which randomization is unavailable. ### Constraints & Assumptions - Only users who belong to a group with an active live session are eligible at send time. - Delivery can fail because of permissions, platform delivery, or stale device tokens. - Hosts and viewers interact, so one user's treatment can affect another user's outcome. - Users may belong to several groups and receive several eligible events. - The team can log eligibility, assignment, send attempts, deliveries when observable, opens, live-session attendance, and downstream app activity. - Do not assume an observational comparison between openers and non-openers is causal. ### Clarifying Questions to Ask - What product goal is the notification supposed to advance: live attendance, group health, retention, or something else? - Who is eligible, and what frequency caps or quiet hours already exist? - Is treatment the notification policy, a single send, or a user-level opt-in? - Can the system randomize at user, group, event, geography, or time-window level? - How much cross-group membership exists, and can a live session serve both treatment and control users? - Which harms are launch blockers, including opt-outs, complaints, host overload, or displacement? ### Part 1: Define Metrics and Estimands Specify the primary causal estimand, the metric hierarchy, and the notification funnel. Explain how you will handle eligibility, failed delivery, repeat sends, and users who never open the notification. #### Hints Assignment is usually safer than opening as the basis of the primary comparison. Separate mechanism diagnostics from the durable product outcome. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2: Design the Randomized Test Choose a randomization unit and explain how the design limits or measures interference. Describe power, duration, analysis, and validity checks. #### Hints Ask whether treatment changes a shared live session. If so, independent user assignments may not produce independent outcomes. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3: Give a Retrospective Alternative If an A/B test cannot be run, propose the strongest feasible historical design. State the identification assumptions, falsification checks, and remaining limitations. #### Hints Look for plausibly exogenous policy, timing, or eligibility variation. Regression adjustment cannot repair every unmeasured difference. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions 1. How would you estimate the effect among users whose notification was actually delivered? 2. When would group-level randomization be preferable to an event-level switchback? 3. What if treatment improves attendance but reduces host satisfaction? 4. How would frequency caps become part of the experiment rather than a fixed assumption? 5. Which result would convince you that the apparent lift is only novelty?