You’re asked to estimate treatment effects with linear regression instead of cohort means. a) Specify an OLS (or GLM) model for contribution per order that includes treatment, daypart, market (Miami vs others), shopper availability index, basket size, and treatment×Sunday×Miami interactions; state identification assumptions required for unbiased ATE and CATE. b) Compare this regression‑adjusted estimator to plain cohort differences in terms of bias, variance, and interpretability when covariates are imbalanced. c) Given three 95% confidence intervals for cohorts (new, repeat‑low‑basket, repeat‑high‑basket), explain what overlap does and does not imply about pairwise significance and about an overall treatment effect with multiple comparisons; propose a proper multiple‑testing correction or a hierarchical model and justify it.

This question evaluates a data scientist's competency in regression-adjusted causal estimation, specifying interaction terms and assessing heterogeneous treatment effects (ATE and CATE) under stated identification assumptions.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Instacart.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Instacart during technical interviews.

Use regression vs cohorts for A/B estimation | Instacart Interview Question

Regression-adjusted estimation of treatment effects for contribution per order

Context

You are analyzing an A/B test at the order level. The outcome is contribution per order (a continuous, possibly skewed, monetary metric). You have the following variables:

Treatment indicator T (1 = treated, 0 = control).
Daypart fixed effects (time-of-day categories).
Market indicator for Miami vs all other markets.
Sunday indicator (1 = Sunday, 0 = other days).
Shopper availability index (continuous, typically at market-day granularity).
Basket size (use a pre-treatment segment or baseline measure; do not use the realized, post-treatment basket size).
A treatment × Sunday × Miami interaction of interest.

Answer the following:

Model and assumptions

Specify an OLS (or GLM) model for contribution per order Y that includes: treatment, daypart, market (Miami vs others), shopper availability index, basket size, and treatment × Sunday × Miami interactions. State the identification assumptions needed for unbiased ATE and CATE.

Regression vs cohort means

Compare the regression-adjusted estimator to plain cohort differences (e.g., by Sunday, Miami, basket cohorts) in terms of bias, variance, and interpretability when covariates are imbalanced.

Confidence intervals and multiple comparisons

You’re given three 95% confidence intervals (CIs) for cohort-specific treatment effects (new, repeat–low-basket, repeat–high-basket). Explain what CI overlap does and does not imply about pairwise significance and about an overall treatment effect across cohorts when facing multiple comparisons. Propose a proper multiple-testing correction or a hierarchical model and justify it.

Context

You are analyzing an A/B test at the order level. The outcome is contribution per order (a continuous, possibly skewed, monetary metric). You have the following variables:

Treatment indicator T (1 = treated, 0 = control).

Daypart fixed effects (time-of-day categories).

Market indicator for Miami vs all other markets.

Sunday indicator (1 = Sunday, 0 = other days).

Shopper availability index (continuous, typically at market-day granularity).

Basket size (use a pre-treatment segment or baseline measure; do not use the realized, post-treatment basket size).

A treatment × Sunday × Miami interaction of interest.

Answer the following:

Model and assumptions

Specify an OLS (or GLM) model for contribution per order Y that includes: treatment, daypart, market (Miami vs others), shopper availability index, basket size, and treatment × Sunday × Miami interactions. State the identification assumptions needed for unbiased ATE and CATE.

Regression vs cohort means

Compare the regression-adjusted estimator to plain cohort differences (e.g., by Sunday, Miami, basket cohorts) in terms of bias, variance, and interpretability when covariates are imbalanced.

Confidence intervals and multiple comparisons

You’re given three 95% confidence intervals (CIs) for cohort-specific treatment effects (new, repeat–low-basket, repeat–high-basket). Explain what CI overlap does and does not imply about pairwise significance and about an overall treatment effect across cohorts when facing multiple comparisons. Propose a proper multiple-testing correction or a hierarchical model and justify it.

Use regression vs cohorts for A/B estimation

Quick Overview

Use regression vs cohorts for A/B estimation

Regression-adjusted estimation of treatment effects for contribution per order

Context

Write your answer

Use regression vs cohorts for A/B estimation

Quick Overview

Use regression vs cohorts for A/B estimation

Regression-adjusted estimation of treatment effects for contribution per order

Context

Write your answer