Evaluate Widget Impact on User Engagement with A/B Testing

Q: Evaluate Widget Impact on User Engagement with A/B Testing

This question evaluates a data scientist's experimental design and causal inference skills, covering A/B test setup, unit of randomization and exposure, metric selection and guardrails, sample size and power calculations, and identification of biases and logging or implementation pitfalls in the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

A company is preparing to roll out a new in‑app recommendation widget and needs evidence that it improves user engagement.

Task

Design an A/B experiment to evaluate the widget’s impact on daily active users (DAU) and session length.

Address the following:

Experiment design: unit of randomization, eligibility/triggering, assignment, and exposure.
Metrics: choose primary metric(s) and guardrail metrics, and explain why.
Sample size and runtime: how you would determine them (include assumptions and formulas).
Risks: potential biases and implementation pitfalls to address.

Hints: Consider metric sensitivity, power calculations, novelty effects, and logging bias.

Evaluate Widget Impact on User Engagement with A/B Testing

A/B Test Design: New In‑App Recommendation Widget

Scenario

Task

Solution

Comments (0)