How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at PayPal.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at PayPal during technical interviews.

Analyze an A/B test and present recommendation

Quick Overview

This question evaluates experimental design, statistical inference, causal estimands, subgroup analysis, data quality checks, and stakeholder-facing communication in the Analytics & Experimentation domain, targeting an applied, intermediate-to-senior data scientist.

You are given an offline take-home style project before an onsite interview. You must analyze an A/B test and present your findings in slides.

Assume you receive a user-level dataset experiment_events with:

user_id (string)
variant (string; 'control' or 'treatment')
assignment_ts (timestamp, UTC)
country (string)
platform (string; 'ios', 'android', 'web')
exposed (boolean; whether the user actually saw the new experience)
orders_7d (int; orders within 7 days after assignment)
revenue_7d (float; revenue within 7 days after assignment)
support_tickets_7d (int)
is_new_user (boolean)

You also have pre-period covariates in user_pre_period:

user_id
orders_28d_pre (int)
revenue_28d_pre (float)

Tasks:

Define the primary metric and guardrails you would use to decide whether to ship.
Perform the core statistical analysis you would do (tests/CI) and list the sanity checks (e.g., SRM, balance).
Explain how you would handle noncompliance ( exposed = false for some assigned users) and which estimand you’d report (ITT vs TOT).
Describe how you would check heterogeneous treatment effects (by platform, country, new vs existing users) without p-hacking.
Outline what your slide deck would contain and how you would communicate uncertainty and next steps to PM/Eng.

Quick Overview

You are given an offline take-home style project before an onsite interview. You must analyze an A/B test and present your findings in slides.

Assume you receive a user-level dataset experiment_events with:

user_id (string)
variant (string; 'control' or 'treatment')
assignment_ts (timestamp, UTC)
country (string)
platform (string; 'ios', 'android', 'web')
exposed (boolean; whether the user actually saw the new experience)
orders_7d (int; orders within 7 days after assignment)
revenue_7d (float; revenue within 7 days after assignment)
support_tickets_7d (int)
is_new_user (boolean)

You also have pre-period covariates in user_pre_period:

user_id
orders_28d_pre (int)
revenue_28d_pre (float)

Tasks:

Define the primary metric and guardrails you would use to decide whether to ship.
Perform the core statistical analysis you would do (tests/CI) and list the sanity checks (e.g., SRM, balance).
Explain how you would handle noncompliance ( exposed = false for some assigned users) and which estimand you’d report (ITT vs TOT).
Describe how you would check heterogeneous treatment effects (by platform, country, new vs existing users) without p-hacking.
Outline what your slide deck would contain and how you would communicate uncertainty and next steps to PM/Eng.

Analyze an A/B test and present recommendation

Quick Overview

Solution

Comments (0)

Analyze an A/B test and present recommendation

Quick Overview

Solution

Comments (0)