A high-impact feature cannot be A/B tested due to policy/infra constraints, but leadership needs a go/no-go decision. Propose a complete analysis plan using Synthetic Control (or justify an alternative). Specify: (a) the treatment unit and donor pool construction, including eligibility/exclusion rules to prevent contamination or anticipation; (b) pre-intervention window length and how you’ll handle seasonality, holidays, macro shocks, and data latency; (c) primary outcome and guardrail metrics, with pre-registered success thresholds and a decision rubric; (d) which predictors to include (outcome lags vs. covariates), weight constraints, and how you’ll tune hyperparameters; (e) diagnostics you will require before trusting effects (pre-period RMSPE targets, in-space and in-time placebo tests, pre/post fit plots), and your inference approach (MSPE ratio/permutation tests, uncertainty bands for pointwise and cumulative effects); (f) sensitivity analyses (leave-one-out donors, alternative windows, augmented/regularized SCM, donor reweighting) and how you’ll bound spillovers; (g) heterogeneity and persistence analyses (subgroups, dynamic effects); (h) what you will do if pre-period fit is poor or donors are scarce; and (i) how the results map to a staged launch, rollback criteria, and post-launch monitoring.

This question evaluates a data scientist's competency in causal inference, time-series analysis, and synthetic control methodology for non-randomized experiments, including donor pool construction, pre/post diagnostics, and sensitivity and heterogeneity assessments.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Reddit.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Reddit during technical interviews.

Design a causal evaluation without A/B testing

Non-Randomized Launch Decision via Synthetic Control: Complete Analysis Plan

You need to make a go/no-go decision for a high-impact feature that cannot be A/B tested due to policy/infrastructure constraints. Assume we can gate the feature to one or a small number of units (e.g., a geography × platform cell) and measure time-series KPIs across comparable units.

Propose a complete analysis plan using Synthetic Control (or justify an alternative if SCM is unsuitable). Address the following:

(a) Treatment unit and donor pool construction

Define the treatment unit precisely.
Specify how you will build the donor pool.
Eligibility/exclusion rules to prevent contamination, spillovers, and anticipation effects.

(b) Pre-intervention window and external factors

Pre-period length and rationale.
How you will handle seasonality, holidays, macro shocks, and data latency/backfill.

Primary outcome and guardrail metrics.
Pre-registered success thresholds and a clear go/no-go decision rule.

(d) Predictors, weights, and tuning

Which predictors to include (outcome lags vs. covariates).
Weight constraints and how you will tune hyperparameters.

(e) Diagnostics and inference

Required diagnostics before trusting effects (e.g., pre-period RMSPE targets, placebo tests in-space/in-time, pre/post fit plots).
Inference approach (MSPE ratio/permutation tests, uncertainty bands for pointwise and cumulative effects).

(f) Sensitivity and spillovers

Sensitivity analyses (leave-one-out donors, alternative windows, augmented/regularized SCM, donor reweighting).
How you will bound and assess spillovers/contamination.

(g) Heterogeneity and persistence

Subgroup and dynamic-effect analyses to assess heterogeneity and persistence/decay.

(h) Fallbacks

What you will do if pre-period fit is poor or donors are scarce.

(i) Launch mapping

How results translate into a staged launch plan, rollback criteria, and post-launch monitoring.

Non-Randomized Launch Decision via Synthetic Control: Complete Analysis Plan

Propose a complete analysis plan using Synthetic Control (or justify an alternative if SCM is unsuitable). Address the following:

(a) Treatment unit and donor pool construction

Define the treatment unit precisely.
Specify how you will build the donor pool.
Eligibility/exclusion rules to prevent contamination, spillovers, and anticipation effects.

(b) Pre-intervention window and external factors

Pre-period length and rationale.
How you will handle seasonality, holidays, macro shocks, and data latency/backfill.

Primary outcome and guardrail metrics.
Pre-registered success thresholds and a clear go/no-go decision rule.

(d) Predictors, weights, and tuning

Which predictors to include (outcome lags vs. covariates).
Weight constraints and how you will tune hyperparameters.

(e) Diagnostics and inference

Required diagnostics before trusting effects (e.g., pre-period RMSPE targets, placebo tests in-space/in-time, pre/post fit plots).
Inference approach (MSPE ratio/permutation tests, uncertainty bands for pointwise and cumulative effects).

(f) Sensitivity and spillovers

Sensitivity analyses (leave-one-out donors, alternative windows, augmented/regularized SCM, donor reweighting).
How you will bound and assess spillovers/contamination.

(g) Heterogeneity and persistence

Subgroup and dynamic-effect analyses to assess heterogeneity and persistence/decay.

(h) Fallbacks

What you will do if pre-period fit is poor or donors are scarce.

(i) Launch mapping

How results translate into a staged launch plan, rollback criteria, and post-launch monitoring.

Design a causal evaluation without A/B testing

Quick Overview

Non-Randomized Launch Decision via Synthetic Control: Complete Analysis Plan

Solution

Comments (0)

Design a causal evaluation without A/B testing

Quick Overview

Non-Randomized Launch Decision via Synthetic Control: Complete Analysis Plan

Solution

Comments (0)