Determine User Demand for New Video-Calling Feature

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine User Demand for New Video-Calling Feature states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Determine User Demand for New Video-Calling Feature

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

##### Scenario A consumer messaging product is preparing to launch an in-app video-calling feature and wants to understand demand, set evaluation metrics, and manage trade-offs. ##### Question Using only the provided SQL dataset, how would you determine whether users actually need or want the new video-calling feature? If you were granted access to all internal data sources, what additional analyses would you perform to gauge demand for the feature? After release, how would you evaluate or measure the quality and success of the video-calling feature? If the primary success metric improves while a non-primary (guardrail) metric declines, how would you proceed and communicate the decision? ##### Hints Consider penetration/activation rates, funnel or cohort analyses, A/B test design, metric hierarchy, guardrails, and follow-up monitoring.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine User Demand for New Video-Calling Feature states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Aug 4, 2025, 10:55 AM
hardData ScientistTechnical ScreenAnalytics & Experimentation
4
0

Determine User Demand for New Video-Calling Feature

Analytics Design: Demand Sizing, Evaluation Metrics, and Trade-offs for a New Video-Calling Feature

Context (Assumptions to Make the Task Self-Contained)

Assume you have access to a SQL dataset from a consumer messaging app with the following representative tables:

  • users(user_id, signup_date, country, device)
  • app_events(user_id, event_name, ts, props)
  • messages(message_id, thread_id, sender_id, recipient_id, sent_at, attachment_type)
  • call_events(call_id, caller_id, callee_id, call_type, started_at, ended_at, status)
    • call_type ∈ {voice} today; video will be added at launch.
    • status ∈ {connected, failed, canceled}.

If the dataset lacks a given table (e.g., voice calls), propose proxy analyses using the tables that exist.

Tasks

  1. Using only the provided SQL dataset, how would you determine whether users need or want video calling?
  2. If you had access to all internal data sources, what additional analyses would you run to gauge demand?
  3. After release, how would you evaluate the quality and success of video calling? Define a metric hierarchy and funnels.
  4. If the primary success metric improves but a guardrail metric declines, how would you proceed and communicate the decision?

Hints

  • Consider penetration/activation rates, funnel or cohort analyses, A/B test design, metric hierarchy, guardrails, and follow-up monitoring.

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...