Analytics Design: Demand Sizing, Evaluation Metrics, and Trade-offs for a New Video-Calling Feature

Context (Assumptions to Make the Task Self-Contained)

Assume you have access to a SQL dataset from a consumer messaging app with the following representative tables:

• users(user_id, signup_date, country, device)
• app_events(user_id, event_name, ts, props)
• messages(message_id, thread_id, sender_id, recipient_id, sent_at, attachment_type)
• call_events(call_id, caller_id, callee_id, call_type, started_at, ended_at, status)
  • call_type ∈ {voice} today; video will be added at launch.
  • status ∈ {connected, failed, canceled}.

If the dataset lacks a given table (e.g., voice calls), propose proxy analyses using the tables that exist.

Tasks

1. Using only the provided SQL dataset, how would you determine whether users need or want video calling?
2. If you had access to all internal data sources, what additional analyses would you run to gauge demand?
3. After release, how would you evaluate the quality and success of video calling? Define a metric hierarchy and funnels.
4. If the primary success metric improves but a guardrail metric declines, how would you proceed and communicate the decision?

Hints

• Consider penetration/activation rates, funnel or cohort analyses, A/B test design, metric hierarchy, guardrails, and follow-up monitoring.