Design an A/B Test for Dashboard Engagement Impact
Quick Overview
Amazon data scientist experimentation prompt on designing an A/B test for an energy-usage dashboard redesign, including hypotheses, metrics, guardrails, power, test duration, seasonality, and analysis methods.
Design an A/B Test for Dashboard Engagement Impact
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Product team is launching a redesigned energy-usage dashboard and needs to measure its impact.
##### Question
How would you design an A/B test to determine whether the new dashboard increases user engagement? Which metrics and statistical methods would you use, and how would you decide the test duration?
##### Hints
State hypothesis, randomize, compute sample size, discuss p-values, guardrail metrics.
Quick Answer: Amazon data scientist experimentation prompt on designing an A/B test for an energy-usage dashboard redesign, including hypotheses, metrics, guardrails, power, test duration, seasonality, and analysis methods.
A product team is launching a redesigned energy-usage dashboard in a consumer app and wants to measure whether the redesign increases user engagement.
Constraints & Assumptions
Randomize at the logged-in user level unless there is a reason to choose household or account-level randomization.
Include only eligible users who can access the dashboard.
Traffic and energy behavior may vary by day of week, season, weather, and billing-cycle timing.
Analyze assigned users using intent-to-treat, and separately inspect exposed users as a diagnostic.
Clarifying Questions to Ask Guidance
What does "engagement" mean for this dashboard: visits, repeat usage, time spent, actions taken, or energy-saving behavior?
Is the redesign purely UI, or does it change recommendations, alerts, or data freshness?
Are users grouped by household/account, and can multiple users share one utility account?
What minimum effect would justify launch?
What a Strong Answer Covers Guidance
Hypotheses, treatment/control definitions, eligibility, randomization unit, exposure logging, and ramp plan.
One primary metric such as weekly dashboard engaged-user rate or meaningful dashboard actions per user.
Secondary metrics such as visits, repeat visits, click-through on insights, alert setup, savings-plan actions, and retention.
Guardrails such as app crashes, latency, support contacts, opt-outs, misinterpretation of energy data, and downstream bill-payment or satisfaction metrics.
Statistical methods: two-sample tests or regression, confidence intervals, p-values, CUPED or covariate adjustment, and multiple-testing control.
Sample size, power, minimum detectable effect, duration covering weekly seasonality, and novelty or learning effects.
Data-quality checks such as SRM, instrumentation validation, balance checks, and pre-registered decision rules.
Follow-up Questions Guidance
What if engagement increases but users take fewer energy-saving actions?
How would seasonality affect test duration?
How would you handle household accounts with multiple users?
