Measuring Notification Quality and Incremental Value for a Travel-Content App

Cycle is an app where members share and browse travel content. It already sends members three kinds of notifications: content recommendation notifications, social notifications and safety notifications. You are asked how good these notifications are and how much value they add. The case starts from the live notification system (Parts 1, 2 and 4); Part 3 asks you to imagine that Cycle had no notifications yet. ### Clarifying Questions - What is Cycle's goal for notifications: more return visits, more posting and commenting, member safety, or a mix that differs by type? - What does a safety notification cover: account security, or safety conditions at a travel destination? Can members turn it off? - What can a member do with a notification besides tapping it: hide it, give negative feedback, or turn off a whole type in settings? - Which channels carry notifications: mobile push, an in-app inbox, email? ### Part 1 — Measure notification quality How would you measure the quality of Cycle's notifications? Click-through rate (CTR) is the natural starting point; what other metrics would you track alongside it, and would you judge all three types by the same metrics? ```hint Three types, three jobs A safety notification that nobody taps may still have done its job. Ask what each type is for before choosing how to judge it. ``` ```hint After the tap, and instead of the tap A tap is only one outcome of a delivered notification. List what else can happen next, good and bad, including what the member finds when the app opens. ``` #### What This Part Should Cover - Metrics tied to the job of each notification type, rather than one CTR for everything - Negative and technical signals alongside engagement - What happens after the tap, and member-level outcomes beyond the notification itself ### Part 2 — Choose the attribution window Suppose CTR counts a tap if it happens within 7 days of the notification being sent. Is a tap 7 days later still a response to the notification? How would you choose the attribution window instead? ```hint Let the delays decide A window is a cut-off on the time between delivery and tap. Think about what you could plot from notifications that were eventually tapped. ``` #### What This Part Should Cover - Why a window that is too long or too short distorts CTR - A data-driven method for choosing the window, and whether it should differ by type or channel - Practical consequences: readout delay, notifications sent near the end of the analysis period, consistency across compared groups ### Part 3 — Measure the incremental value of notifications Now suppose Cycle has no notification system and is about to launch one. How would you measure the incremental value of notifications? If you would compare engagement before and after the launch, explain what that comparison can and cannot show. If you would run an A/B test, give the randomization unit and the metrics you would compare. Members in the control group receive no notifications. ```hint Can you compute it for control? For each metric you propose, check whether it can be computed for a member who never receives a notification. ``` ```hint The launch date A before-and-after comparison credits the launch with everything else that changed over the same weeks. Think about what changes over a few weeks in a travel product. ``` #### What This Part Should Cover - The weakness of a before/after comparison and the stronger randomized design - A primary metric that exists in both groups and reflects the product goal, with secondary metrics and guardrails - Randomization unit, the handling of safety notifications, interference between members, power and duration ### Part 4 — CTR and hide rate both rise Suppose a later test changes the notifications, with both groups receiving notifications, and treatment shows a higher CTR and a higher hide rate than control. What do you do, and how do you decide whether to roll the change out? ```hint Same members or different ones? Ask whether the extra taps and the extra hides come from the same members, the same notification type or the same notifications. ``` #### What This Part Should Cover - Checking that both movements are real and not artifacts of volume or instrumentation - Decomposing the result by notification type and member segment - A decision based on member-level value and the long-term cost of hides and opt-outs ### What a Strong Answer Covers - Clarifies Cycle's product goal and the purpose of each notification type before choosing metrics - Distinguishes notification-level metrics from member-level outcomes, and knows which kind decides a launch - Chooses windows and definitions from data rather than habit - A valid causal design with a primary metric defined for both arms - A principled way to resolve conflicting metrics, with segments and long-term effects in view ### Follow-up Questions - If members can unsubscribe from each notification type, how would you use unsubscribe rates, and how do they relate to the hide rate? - Should safety notifications be withheld from a control group? If not, how would you measure their value? - Social notifications are triggered by other members' actions. How could treatment members' activity leak into the control group, and what would that do to your estimate? - How would you detect notification fatigue that builds up over months, after a short test has ended?