Push Notification Quality: Metric, Baseline Assessment, and Experiment Design

Background

A mobile app uses push notifications to drive user engagement. The company wants to send only high-quality notifications and to continuously evaluate and improve its notification strategy.

Assume the following data are available:

• Event logs for push notifications with fields: user_id, device_os, locale, user_tenure, activity cohort, notification_type/template, send_time, delivered/viewed/clicked flags, session starts, dwell time, opt-out/mute/uninstall events, and conversions within a configurable window (e.g., 24 hours post-send).
• Historical click-through-rate (CTR) by segment and time (e.g., by notification type, hour-of-day, day-of-week, OS, locale, user cohort).
• Ability to run A/B tests at the user level or trigger level.

Tasks

1. Propose and justify a metric that captures the quality of notifications beyond raw CTR.
2. Given historical CTR data, explain how to determine whether current CTR is good or bad, accounting for seasonality and mix shifts.
3. Design an A/B test for a new notification strategy (e.g., targeting, send-time, or copy). Specify:
   • Experiment setup and randomization unit
   • Success metrics and guardrails
   • Sample sizing/power, duration, and segmentation
   • Analysis plan and validity checks
   • Post-test analysis and next steps