##### Scenario Evaluating the health of a new product feature after launch. ##### Question What core metrics would you monitor and how are they computed? A key metric suddenly drops 5%. List possible root causes and the data you would pull to validate each. Design an experiment to test a UI change intended to improve the metric (units, hypothesis, assignment, duration, success criteria). ##### Hints Think DAU/WAU, conversion funnel, exposure units, power calculations, guardrail metrics.

This question evaluates competency in product analytics, metric definition, diagnostic troubleshooting, and experimentation design within the Analytics & Experimentation domain for data scientists.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Robinhood.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Robinhood during technical interviews.

Evaluate Core Metrics for New Product Feature Launch

Scenario

You are a data scientist evaluating the health of a newly launched product feature in a consumer-facing app (e.g., investing/finance). The goal is to define what to monitor, diagnose issues if a key metric drops, and design an experiment to improve performance.

Tasks

Core Metrics and Computation
- List the core metrics you would monitor post-launch and how each is computed.
Investigation of a 5% Drop
- A key metric suddenly drops by 5% (assume a relative drop unless stated otherwise). List plausible root causes and specify the exact data you would pull to validate or rule out each cause.
Experiment Design for a UI Change
- Design an experiment to test a UI change intended to improve the key metric. Specify:
  - Exposure unit and eligibility
  - Primary/secondary metrics and clear hypotheses
  - Randomization/assignment strategy
  - Duration and power/MDE assumptions
  - Success criteria and guardrail metrics

Hints: Think DAU/WAU and stickiness, conversion funnel definitions and denominators, exposure units, power calculations, and guardrail metrics (e.g., stability, latency, error rates, revenue risk).

Tasks

Core Metrics and Computation

List the core metrics you would monitor post-launch and how each is computed.

Investigation of a 5% Drop

A key metric suddenly drops by 5% (assume a relative drop unless stated otherwise). List plausible root causes and specify the exact data you would pull to validate or rule out each cause.

Experiment Design for a UI Change

Design an experiment to test a UI change intended to improve the key metric. Specify:
- Exposure unit and eligibility
- Primary/secondary metrics and clear hypotheses
- Randomization/assignment strategy
- Duration and power/MDE assumptions
- Success criteria and guardrail metrics

Hints: Think DAU/WAU and stickiness, conversion funnel definitions and denominators, exposure units, power calculations, and guardrail metrics (e.g., stability, latency, error rates, revenue risk).

Evaluate Core Metrics for New Product Feature Launch

Quick Overview

Scenario

Tasks

Solution

Comments (0)

Evaluate Core Metrics for New Product Feature Launch

Quick Overview

Scenario

Tasks

Solution

Comments (0)