You run a user-level 50/50 A/B test on August 1, 2025 for a new messaging feature; the primary metric is average 14-day engagement minutes per assigned user. Issues observed: (1) Noncompliance: 20% of Treatment users never see the feature; 5% of Control users get it via leakage. (2) Missing data: 8% of sessions fail to log due to a bug over-represented on Android 13; conditional on device and region, logging is independent of treatment. (3) Interference: viral invites cause 10% of Control users to interact directly with treated users. (4) Early stopping: the test was ended on day 7 after observing p < 0.05, even though the metric was defined over 14 days. For the difference-in-means of assigned groups, answer: (a) State the precise assumptions under which this estimator is unbiased for the ITT (e.g., randomization, SUTVA/no interference, consistency, positivity, correct handling of missingness). (b) For each issue (1)–(4), say whether the ITT estimator remains unbiased; if bias arises, state its likely direction and why. (c) Give a concrete analysis plan to recover an unbiased or approximately unbiased estimate of the treatment effect on 14-day engagement: include how you would handle noncompliance (e.g., report ITT; optionally estimate TOT via IV with assignment as the instrument and state required assumptions), missing logs (e.g., IPW or multiple imputation using device/region strata), interference (e.g., cluster-level or exposure-based adjustment/sensitivity analysis), and early stopping (e.g., pre-specified alpha spending or sequential corrections). (d) List specific diagnostic checks you would run to support the assumptions (balance checks, missingness patterns by strata, spillover detection, and robustness/sensitivity analyses).

This question evaluates a data scientist's ability to diagnose unbiasedness of an intent-to-treat (ITT) estimator in A/B testing under noncompliance, missing logs, interference, and early stopping while reasoning about causal assumptions and potential bias direction.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Google.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Google during technical interviews.

Diagnose unbiasedness in a messy A/B test | Google Interview Question

Q: Diagnose unbiasedness in a messy A/B test

This question evaluates a data scientist's ability to diagnose unbiasedness of an intent-to-treat (ITT) estimator in A/B testing under noncompliance, missing logs, interference, and early stopping while reasoning about causal assumptions and potential bias direction.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Google.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Google during technical interviews.

A/B Test ITT Unbiasedness and Remedies Under Noncompliance, Missing Logs, Interference, and Early Stopping

Setup

Design: User-level 50/50 A/B test starting Aug 1, 2025.
Primary metric: Average 14-day engagement minutes per assigned user (users with zero usage contribute 0).
Observed issues:
1. Noncompliance: 20% of Treatment never see the feature; 5% of Control get it (leakage).
2. Missing data: 8% of sessions fail to log due to a bug over-represented on Android 13; conditional on device and region, logging is independent of treatment.
3. Interference: Viral invites cause 10% of Control users to interact directly with treated users.
4. Early stopping: Stopped on day 7 after observing p < 0.05, even though the metric is defined over 14 days.

Let Z ∈ {0,1} denote assignment (0=Control, 1=Treatment). The estimator of interest is the difference in mean observed outcomes by assignment, i.e., E[Ŷ | Z=1] − E[Ŷ | Z=0], where Ŷ is the measured 14-day minutes.

Tasks

(a) State the precise assumptions under which this estimator is unbiased for the intent-to-treat (ITT) effect on 14-day engagement.

(b) For each issue (1)–(4), say whether the ITT estimator remains unbiased; if bias arises, state its likely direction and why.

(c) Propose a concrete analysis plan to recover an unbiased or approximately unbiased estimate of the treatment effect on 14-day engagement. Include handling of:

Noncompliance (e.g., report ITT; optionally estimate TOT via IV with assignment as instrument and list assumptions),
Missing logs (e.g., IPW or multiple imputation using device/region strata),
Interference (e.g., cluster-level or exposure-based adjustment/sensitivity analysis),
Early stopping (e.g., pre-specified alpha spending or sequential corrections).

(d) List specific diagnostic checks to support assumptions (balance checks, missingness patterns by strata, spillover detection, robustness/sensitivity analyses).

A/B Test ITT Unbiasedness and Remedies Under Noncompliance, Missing Logs, Interference, and Early Stopping

Setup

Design: User-level 50/50 A/B test starting Aug 1, 2025.
Primary metric: Average 14-day engagement minutes per assigned user (users with zero usage contribute 0).
Observed issues:
1. Noncompliance: 20% of Treatment never see the feature; 5% of Control get it (leakage).
2. Missing data: 8% of sessions fail to log due to a bug over-represented on Android 13; conditional on device and region, logging is independent of treatment.
3. Interference: Viral invites cause 10% of Control users to interact directly with treated users.
4. Early stopping: Stopped on day 7 after observing p < 0.05, even though the metric is defined over 14 days.

Tasks

(a) State the precise assumptions under which this estimator is unbiased for the intent-to-treat (ITT) effect on 14-day engagement.

(b) For each issue (1)–(4), say whether the ITT estimator remains unbiased; if bias arises, state its likely direction and why.

(c) Propose a concrete analysis plan to recover an unbiased or approximately unbiased estimate of the treatment effect on 14-day engagement. Include handling of:

Noncompliance (e.g., report ITT; optionally estimate TOT via IV with assignment as instrument and list assumptions),
Missing logs (e.g., IPW or multiple imputation using device/region strata),
Interference (e.g., cluster-level or exposure-based adjustment/sensitivity analysis),
Early stopping (e.g., pre-specified alpha spending or sequential corrections).

(d) List specific diagnostic checks to support assumptions (balance checks, missingness patterns by strata, spillover detection, robustness/sensitivity analyses).

Diagnose unbiasedness in a messy A/B test

Quick Overview

A/B Test ITT Unbiasedness and Remedies Under Noncompliance, Missing Logs, Interference, and Early Stopping

Setup

Tasks

Solution

Submit Your Answer to Earn 20XP

Diagnose unbiasedness in a messy A/B test

Quick Overview

A/B Test ITT Unbiasedness and Remedies Under Noncompliance, Missing Logs, Interference, and Early Stopping

Setup

Tasks

Solution

Submit Your Answer to Earn 20XP