Design evaluation when A/B test is impossible

Scenario

You want to evaluate whether a product or model change (e.g., a new ranking strategy, pricing rule, or UI change) improves business outcomes.

However, you cannot run a standard randomized A/B test due to one or more constraints:

• Legal/compliance restrictions (cannot randomize users)
• Platform limitation (no experimentation framework)
• Strong network effects / interference (user outcomes affect each other)
• Rollout must be global (no holdout allowed)
• Treatment is self-selected (users opt in)

Questions

1. What metrics would you choose?
   • Propose a primary metric and at least 2 diagnostic and 2 guardrail metrics.
   • Explain tradeoffs (e.g., short-term vs long-term, sensitivity vs robustness).
2. How would you estimate the counterfactual (what would have happened without the change)?
   • Propose multiple causal inference approaches (at least 3), e.g. matching/weighting, difference-in-differences, synthetic control, regression discontinuity, instrumental variables, uplift modeling, etc.
   • For each approach, state key assumptions , what data you need, and how you would validate/pressure-test the assumptions.
3. How would you handle common pitfalls ?
   • Confounding / selection bias
   • Seasonality and time trends
   • Delayed effects / novelty effects
   • Spillovers/interference
   • Missing data and metric instrumentation changes
4. Product monitoring follow-up:
   • Suppose after launch, the core KPI drops sharply on a single day . Outline a structured investigation plan to determine whether it’s (a) a real product issue, (b) a logging/pipeline issue, or (c) an external shock.
   • Include what slices you would check first and what “sanity checks” you would run.