##### Scenario A consumer messaging product is preparing to launch an in-app video-calling feature and wants to understand demand, set evaluation metrics, and manage trade-offs. ##### Question Using only the provided SQL dataset, how would you determine whether users actually need or want the new video-calling feature? If you were granted access to all internal data sources, what additional analyses would you perform to gauge demand for the feature? After release, how would you evaluate or measure the quality and success of the video-calling feature? If the primary success metric improves while a non-primary (guardrail) metric declines, how would you proceed and communicate the decision? ##### Hints Consider penetration/activation rates, funnel or cohort analyses, A/B test design, metric hierarchy, guardrails, and follow-up monitoring.

This question evaluates a data scientist's competency in analytics design, demand sizing, metric hierarchy construction, experimentation and trade-off analysis for a new product feature.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Determine User Demand for New Video-Calling Feature

Analytics Design: Demand Sizing, Evaluation Metrics, and Trade-offs for a New Video-Calling Feature

Context (Assumptions to Make the Task Self-Contained)

Assume you have access to a SQL dataset from a consumer messaging app with the following representative tables:

users(user_id, signup_date, country, device)
app_events(user_id, event_name, ts, props)
messages(message_id, thread_id, sender_id, recipient_id, sent_at, attachment_type)
call_events(call_id, caller_id, callee_id, call_type, started_at, ended_at, status)
- call_type ∈ {voice} today; video will be added at launch.
- status ∈ {connected, failed, canceled}.

If the dataset lacks a given table (e.g., voice calls), propose proxy analyses using the tables that exist.

Tasks

Using only the provided SQL dataset, how would you determine whether users need or want video calling?
If you had access to all internal data sources, what additional analyses would you run to gauge demand?
After release, how would you evaluate the quality and success of video calling? Define a metric hierarchy and funnels.
If the primary success metric improves but a guardrail metric declines, how would you proceed and communicate the decision?

Hints

Consider penetration/activation rates, funnel or cohort analyses, A/B test design, metric hierarchy, guardrails, and follow-up monitoring.

Context (Assumptions to Make the Task Self-Contained)

Assume you have access to a SQL dataset from a consumer messaging app with the following representative tables:

users(user_id, signup_date, country, device)

app_events(user_id, event_name, ts, props)

messages(message_id, thread_id, sender_id, recipient_id, sent_at, attachment_type)

call_events(call_id, caller_id, callee_id, call_type, started_at, ended_at, status)

call_type ∈ {voice} today; video will be added at launch.
status ∈ {connected, failed, canceled}.

If the dataset lacks a given table (e.g., voice calls), propose proxy analyses using the tables that exist.

Tasks

Using only the provided SQL dataset, how would you determine whether users need or want video calling?

If you had access to all internal data sources, what additional analyses would you run to gauge demand?

After release, how would you evaluate the quality and success of video calling? Define a metric hierarchy and funnels.

If the primary success metric improves but a guardrail metric declines, how would you proceed and communicate the decision?

Determine User Demand for New Video-Calling Feature

Quick Overview

Analytics Design: Demand Sizing, Evaluation Metrics, and Trade-offs for a New Video-Calling Feature

Context (Assumptions to Make the Task Self-Contained)

Tasks

Hints

Solution

Submit Your Answer to Earn 20XP

Determine User Demand for New Video-Calling Feature

Quick Overview

Analytics Design: Demand Sizing, Evaluation Metrics, and Trade-offs for a New Video-Calling Feature

Context (Assumptions to Make the Task Self-Contained)

Tasks

Hints

Solution

Submit Your Answer to Earn 20XP