##### Scenario Product team is launching a redesigned energy-usage dashboard and needs to measure its impact. ##### Question How would you design an A/B test to determine whether the new dashboard increases user engagement? Which metrics and statistical methods would you use, and how would you decide the test duration? ##### Hints State hypothesis, randomize, compute sample size, discuss p-values, guardrail metrics.

This question evaluates a data scientist's competency in experimental design, statistical inference, metric selection, and instrumentation for product analytics.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Amazon.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Amazon during technical interviews.

Design an A/B Test for Dashboard Engagement Impact

Redesign Experiment: Energy-Usage Dashboard

Context

A redesigned energy-usage dashboard is being launched in a consumer app. The goal is to measure whether the redesign increases user engagement with the dashboard.

Assume:

Unit of randomization is the user (logged-in ID).
Only eligible, active users who can see the dashboard are included.
Traffic and seasonality vary by day of week.

Task

Design an A/B test to determine whether the new dashboard increases user engagement.

Address the following:

State clear hypotheses and define the unit of randomization and exposure.
Specify primary and secondary metrics, plus guardrail metrics.
Describe the statistical methods for analysis (p-values, confidence intervals, variance reduction).
Compute/justify sample size and determine test duration.
Include practical checks (randomization, instrumentation, novelty/learning effects).

Redesign Experiment: Energy-Usage Dashboard

Context

A redesigned energy-usage dashboard is being launched in a consumer app. The goal is to measure whether the redesign increases user engagement with the dashboard.

Assume:

Unit of randomization is the user (logged-in ID).
Only eligible, active users who can see the dashboard are included.
Traffic and seasonality vary by day of week.

Task

Design an A/B test to determine whether the new dashboard increases user engagement.

Address the following:

State clear hypotheses and define the unit of randomization and exposure.
Specify primary and secondary metrics, plus guardrail metrics.
Describe the statistical methods for analysis (p-values, confidence intervals, variance reduction).
Compute/justify sample size and determine test duration.
Include practical checks (randomization, instrumentation, novelty/learning effects).

Design an A/B Test for Dashboard Engagement Impact

Quick Overview

Redesign Experiment: Energy-Usage Dashboard

Context

Task

Solution

Submit Your Answer to Earn 20XP

Design an A/B Test for Dashboard Engagement Impact

Quick Overview

Redesign Experiment: Energy-Usage Dashboard

Context

Task

Solution

Submit Your Answer to Earn 20XP