Evaluate Core Metrics for New Product Feature Launch

Q: Evaluate Core Metrics for New Product Feature Launch

This question evaluates competency in product analytics, metric definition, diagnostic troubleshooting, and experimentation design within the Analytics & Experimentation domain for data scientists.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

You are a data scientist evaluating the health of a newly launched product feature in a consumer-facing app (e.g., investing/finance). The goal is to define what to monitor, diagnose issues if a key metric drops, and design an experiment to improve performance.

Tasks

Core Metrics and Computation
- List the core metrics you would monitor post-launch and how each is computed.
Investigation of a 5% Drop
- A key metric suddenly drops by 5% (assume a relative drop unless stated otherwise). List plausible root causes and specify the exact data you would pull to validate or rule out each cause.
Experiment Design for a UI Change
- Design an experiment to test a UI change intended to improve the key metric. Specify:
  - Exposure unit and eligibility
  - Primary/secondary metrics and clear hypotheses
  - Randomization/assignment strategy
  - Duration and power/MDE assumptions
  - Success criteria and guardrail metrics

Hints: Think DAU/WAU and stickiness, conversion funnel definitions and denominators, exposure units, power calculations, and guardrail metrics (e.g., stability, latency, error rates, revenue risk).

Evaluate Core Metrics for New Product Feature Launch

Scenario

Tasks

Solution

Comments (0)