##### Scenario A mobile app team wants to roll out a new recommendation algorithm and needs an A/B test to decide whether to launch it. ##### Question Describe end-to-end how you would design and run this A/B test. What metrics would you track and how would you define success? How do you determine the required sample size and test duration? Name common pitfalls in A/B testing and how to avoid them. ##### Hints Discuss randomization, segmentation, statistical power, guard-rail metrics, stopping rules, and launch criteria.

This question evaluates a data scientist's competency in experimental design, causal inference, instrumentation, metric design, and statistical analysis for online A/B testing.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Chime.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Chime during technical interviews.

Design an Effective A/B Test for Algorithm Launch

Design an A/B Test for a New Mobile App Recommendation Algorithm

Context

A mobile app team plans to ship a new recommendation algorithm that ranks content in the app. Assume:

We can randomize at the user level with sticky assignment.
Traffic is sufficient to run a 50/50 A/B split after a brief ramp.
The algorithm may change engagement and performance (e.g., latency).

Task

Describe, end-to-end, how you would design and run this A/B test:

Experiment design
- Unit of randomization, assignment, segmentation, and ramp plan.
- Exposure definition and logging/instrumentation.
- Analysis plan and stopping rules.
Metrics and success criteria
- Primary, secondary, and guard-rail metrics.
- How you will define success and launch criteria.
Sample size and test duration
- How to determine required sample size and duration.
- Include formulas and a small numeric example.
Common pitfalls and mitigations
- Discuss randomization, segmentation, statistical power, guard-rail metrics, stopping rules, and launch criteria.

Context

A mobile app team plans to ship a new recommendation algorithm that ranks content in the app. Assume:

We can randomize at the user level with sticky assignment.

Traffic is sufficient to run a 50/50 A/B split after a brief ramp.

The algorithm may change engagement and performance (e.g., latency).

Task

Describe, end-to-end, how you would design and run this A/B test:

Experiment design

Unit of randomization, assignment, segmentation, and ramp plan.
Exposure definition and logging/instrumentation.
Analysis plan and stopping rules.

Metrics and success criteria

Primary, secondary, and guard-rail metrics.
How you will define success and launch criteria.

Sample size and test duration

How to determine required sample size and duration.
Include formulas and a small numeric example.

Common pitfalls and mitigations

Discuss randomization, segmentation, statistical power, guard-rail metrics, stopping rules, and launch criteria.

Design an Effective A/B Test for Algorithm Launch

Quick Overview

Design an A/B Test for a New Mobile App Recommendation Algorithm

Context

Task

Solution

Comments (0)

Design an Effective A/B Test for Algorithm Launch

Quick Overview

Design an A/B Test for a New Mobile App Recommendation Algorithm

Context

Task

Solution

Comments (0)