Design A/B test for AI chat box

Q: Design A/B test for AI chat box

This question evaluates skills in experimental design, product analytics, metrics selection, and causal inference relevant to a Data Scientist role in the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Loading...

Context

A social platform (e.g., Reddit) plans to launch an AI chat box that answers user questions. The functionality overlaps with the existing Search experience.

Task

Design an A/B test to evaluate the launch.

Requirements

Define the primary metric , diagnostic metrics , and guardrail metrics .
Describe the experiment design :
- unit of randomization (user/session/etc.)
- eligibility and ramp plan
- experiment duration and how to handle novelty effects
Identify key product impacts and risks , including cannibalization of Search.
List major confounders / biases and how you would mitigate them.
Specify what decision you would make based on possible outcomes.

Design A/B test for AI chat box

Context

Task

Requirements

Solution

Comments (0)

Design A/B test for AI chat box

Overview

Context

Task

Requirements

Solution

Comments (0)