Evaluate Overlapping Shelf Ranking Experiments

Quick Overview

This question evaluates competency in experimental design and causal inference for A/B testing, focusing on handling overlapping traffic, interaction effects between concurrent experiments, metric selection, guardrails, and statistical validity.

Evaluate Overlapping Shelf Ranking Experiments

Company: Fetch

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

Fetch is running two A/B experiments in its mobile app. ### Product context The app displays product shelves. Each shelf contains multiple products, and the app also decides the order in which shelves are shown to users. ### Experiments 1. **Experiment 1: Product ordering within a shelf** - Treatment: change the order of products inside each shelf. - Traffic allocation: 50% of app traffic. 2. **Experiment 2: Shelf ordering in the app** - Treatment: change the order of shelves shown in the app. - Traffic allocation: 9% of app traffic. Assume the primary business goal is to improve user engagement and shopping behavior, such as product clicks, add-to-cart events, purchases, revenue, or receipt submissions. ### Questions 1. What should you consider when designing and analyzing these two A/B experiments? 2. If traffic cannot be perfectly isolated between the two experiments, are the experiment results completely unusable? Why or why not? 3. If the two experiments run with overlapping traffic, how would you measure the impact of Experiment 2 on Experiment 1? 4. What metrics, guardrails, statistical checks, and interpretation risks would you include in your analysis?

Quick Answer: This question evaluates competency in experimental design and causal inference for A/B testing, focusing on handling overlapping traffic, interaction effects between concurrent experiments, metric selection, guardrails, and statistical validity.

|Home/Analytics & Experimentation/Fetch
Fetch logo
Fetch
Apr 27, 2026, 12:00 AM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
2
0

Fetch is running two A/B experiments in its mobile app.

Product context

The app displays product shelves. Each shelf contains multiple products, and the app also decides the order in which shelves are shown to users.

Experiments

  1. Experiment 1: Product ordering within a shelf
    • Treatment: change the order of products inside each shelf.
    • Traffic allocation: 50% of app traffic.
  2. Experiment 2: Shelf ordering in the app
    • Treatment: change the order of shelves shown in the app.
    • Traffic allocation: 9% of app traffic.

Assume the primary business goal is to improve user engagement and shopping behavior, such as product clicks, add-to-cart events, purchases, revenue, or receipt submissions.

Questions

  1. What should you consider when designing and analyzing these two A/B experiments?
  2. If traffic cannot be perfectly isolated between the two experiments, are the experiment results completely unusable? Why or why not?
  3. If the two experiments run with overlapping traffic, how would you measure the impact of Experiment 2 on Experiment 1?
  4. What metrics, guardrails, statistical checks, and interpretation risks would you include in your analysis?
Loading comments...