##### Scenario Mobile app sends push notifications and wants to ensure only high-quality notifications are shipped. ##### Question Propose and justify a metric that captures high-quality notifications. Given historical click-through-rate (CTR) data, how would you determine whether the current CTR is good or bad? Design an A/B test for a new notification strategy; explain experiment setup, success metrics, and how you would analyze the results in depth. ##### Hints Cover metric formulation, baselines, segments, statistical power, validity threats, and post-test analysis.

This question evaluates a data scientist's competency in metric design, segment-level and time-series analysis, and experimentation including A/B test setup, power calculation, and post-test inference.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Determine High-Quality Notifications with CTR Analysis

Push Notification Quality: Metric, Baseline Assessment, and Experiment Design

Background

A mobile app uses push notifications to drive user engagement. The company wants to send only high-quality notifications and to continuously evaluate and improve its notification strategy.

Assume the following data are available:

Event logs for push notifications with fields: user_id, device_os, locale, user_tenure, activity cohort, notification_type/template, send_time, delivered/viewed/clicked flags, session starts, dwell time, opt-out/mute/uninstall events, and conversions within a configurable window (e.g., 24 hours post-send).
Historical click-through-rate (CTR) by segment and time (e.g., by notification type, hour-of-day, day-of-week, OS, locale, user cohort).
Ability to run A/B tests at the user level or trigger level.

Tasks

Propose and justify a metric that captures the quality of notifications beyond raw CTR.
Given historical CTR data, explain how to determine whether current CTR is good or bad, accounting for seasonality and mix shifts.
Design an A/B test for a new notification strategy (e.g., targeting, send-time, or copy). Specify:
- Experiment setup and randomization unit
- Success metrics and guardrails
- Sample sizing/power, duration, and segmentation
- Analysis plan and validity checks
- Post-test analysis and next steps

Background

A mobile app uses push notifications to drive user engagement. The company wants to send only high-quality notifications and to continuously evaluate and improve its notification strategy.

Assume the following data are available:

Event logs for push notifications with fields: user_id, device_os, locale, user_tenure, activity cohort, notification_type/template, send_time, delivered/viewed/clicked flags, session starts, dwell time, opt-out/mute/uninstall events, and conversions within a configurable window (e.g., 24 hours post-send).

Historical click-through-rate (CTR) by segment and time (e.g., by notification type, hour-of-day, day-of-week, OS, locale, user cohort).

Ability to run A/B tests at the user level or trigger level.

Tasks

Propose and justify a metric that captures the quality of notifications beyond raw CTR.

Given historical CTR data, explain how to determine whether current CTR is good or bad, accounting for seasonality and mix shifts.

Design an A/B test for a new notification strategy (e.g., targeting, send-time, or copy). Specify:

Experiment setup and randomization unit
Success metrics and guardrails
Sample sizing/power, duration, and segmentation
Analysis plan and validity checks
Post-test analysis and next steps

Determine High-Quality Notifications with CTR Analysis

Quick Overview

Push Notification Quality: Metric, Baseline Assessment, and Experiment Design

Background

Tasks

Solution

Submit Your Answer to Earn 20XP

Determine High-Quality Notifications with CTR Analysis

Quick Overview

Push Notification Quality: Metric, Baseline Assessment, and Experiment Design

Background

Tasks

Solution

Submit Your Answer to Earn 20XP