Determine High-Quality Notifications with CTR Analysis

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine High-Quality Notifications with CTR Analysis states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Determine High-Quality Notifications with CTR Analysis

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

##### Scenario Mobile app sends push notifications and wants to ensure only high-quality notifications are shipped. ##### Question Propose and justify a metric that captures high-quality notifications. Given historical click-through-rate (CTR) data, how would you determine whether the current CTR is good or bad? Design an A/B test for a new notification strategy; explain experiment setup, success metrics, and how you would analyze the results in depth. ##### Hints Cover metric formulation, baselines, segments, statistical power, validity threats, and post-test analysis.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine High-Quality Notifications with CTR Analysis states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Aug 4, 2025, 10:55 AM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
5
0

Determine High-Quality Notifications with CTR Analysis

Push Notification Quality: Metric, Baseline Assessment, and Experiment Design

Background

A mobile app uses push notifications to drive user engagement. The company wants to send only high-quality notifications and to continuously evaluate and improve its notification strategy.

Assume the following data are available:

  • Event logs for push notifications with fields: user_id, device_os, locale, user_tenure, activity cohort, notification_type/template, send_time, delivered/viewed/clicked flags, session starts, dwell time, opt-out/mute/uninstall events, and conversions within a configurable window (e.g., 24 hours post-send).
  • Historical click-through-rate (CTR) by segment and time (e.g., by notification type, hour-of-day, day-of-week, OS, locale, user cohort).
  • Ability to run A/B tests at the user level or trigger level.

Tasks

  1. Propose and justify a metric that captures the quality of notifications beyond raw CTR.
  2. Given historical CTR data, explain how to determine whether current CTR is good or bad, accounting for seasonality and mix shifts.
  3. Design an A/B test for a new notification strategy (e.g., targeting, send-time, or copy). Specify:
    • Experiment setup and randomization unit
    • Success metrics and guardrails
    • Sample sizing/power, duration, and segmentation
    • Analysis plan and validity checks
    • Post-test analysis and next steps

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation