How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at TikTok.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at TikTok during technical interviews.

Determine Metrics for Evaluating Homepage Recommendation Carousel

Quick Overview

This question evaluates a data scientist's competency in designing and analyzing A/B experiments for a homepage recommendation carousel, including selection of primary and secondary metrics, assessment of statistical power and practical significance, and the setup of guardrails.

Experiment Evaluation: Homepage Recommendation Carousel

Scenario

A product team has shipped an A/B test for a new recommendation carousel placed on the app's homepage. Users are randomly assigned at the user level to Control (no carousel) or Treatment (homepage shows the carousel). The goal is to improve user engagement and downstream conversions without harming overall experience or performance.

Task

Propose the primary metric(s) and secondary metric(s) you would track to evaluate the experiment.
Describe a clear decision framework to determine whether to roll the feature out to all users, including how you will handle statistical power, practical significance, and guardrails.

Assume a standard 50/50 split, sticky assignment, and at least one weekly cycle of traffic.

Experiment Evaluation: Homepage Recommendation Carousel

Scenario

Task

Propose the primary metric(s) and secondary metric(s) you would track to evaluate the experiment.
Describe a clear decision framework to determine whether to roll the feature out to all users, including how you will handle statistical power, practical significance, and guardrails.

Assume a standard 50/50 split, sticky assignment, and at least one weekly cycle of traffic.

Determine Metrics for Evaluating Homepage Recommendation Carousel

Quick Overview

Experiment Evaluation: Homepage Recommendation Carousel

Scenario

Task

Solution

Submit Your Answer to Earn 20XP

Determine Metrics for Evaluating Homepage Recommendation Carousel

Quick Overview

Experiment Evaluation: Homepage Recommendation Carousel

Scenario

Task

Solution

Submit Your Answer to Earn 20XP