Design an A/B Test for Dashboard Engagement Impact

Q: Design an A/B Test for Dashboard Engagement Impact

This question evaluates a data scientist's competency in experimental design, statistical inference, metric selection, and instrumentation for product analytics.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Redesign Experiment: Energy-Usage Dashboard

Context

A redesigned energy-usage dashboard is being launched in a consumer app. The goal is to measure whether the redesign increases user engagement with the dashboard.

Assume:

Unit of randomization is the user (logged-in ID).
Only eligible, active users who can see the dashboard are included.
Traffic and seasonality vary by day of week.

Task

Design an A/B test to determine whether the new dashboard increases user engagement.

Address the following:

State clear hypotheses and define the unit of randomization and exposure.
Specify primary and secondary metrics, plus guardrail metrics.
Describe the statistical methods for analysis (p-values, confidence intervals, variance reduction).
Compute/justify sample size and determine test duration.
Include practical checks (randomization, instrumentation, novelty/learning effects).

Design an A/B Test for Dashboard Engagement Impact

Redesign Experiment: Energy-Usage Dashboard

Context

Task

Solution

Comments (0)