Homepage Experiment Without a Control Group
A social-media homepage team is analyzing a personalized feed. An intern accidentally launched a treatment to a user cohort without a randomized control group. You have event logs, impression and click events, pre-period user behavior, device, geo, app version, and eligibility flags.
Answer the tasks below as if you are advising whether the results can still support a product decision.
Constraints & Assumptions
-
Acknowledge that causal certainty is weaker without randomized control.
-
Use pre-treatment data only for matching, weighting, and diagnostics.
-
Check overlap, balance, logging quality, and maturation before interpreting treatment effects.
-
Separate exploratory directional evidence from a launch-grade causal estimate.
Clarifying Questions to Ask
-
Why did the cohort receive treatment, and were untreated users eligible under the same rules?
-
Is there a clean pre-period and post-period for every user?
-
Which metric is the decision metric: CTR, retention, DAU, time spent, module engagement, or revenue?
-
Can we rerun the experiment with a proper randomized holdout?
Part 1 - Estimate Impact Without a Randomized Control
Compare matching and propensity-score weighting for estimating treatment impact.
What This Part Should Cover
-
Define treated and candidate comparison users using eligibility and pre-period covariates.
-
Explain exact matching, nearest-neighbor matching, or coarsened matching and when each is useful.
-
Explain propensity-score weighting, overlap, trimming, and balance diagnostics.
-
State assumptions such as conditional ignorability, positivity, and no interference.
-
Recommend sensitivity checks and, if possible, a future randomized experiment.
Part 2 - A/B Test Hygiene Review
List common randomized experiment pitfalls that could bias an existing A/B test.
What This Part Should Cover
-
Check sample ratio mismatch, assignment stickiness, logging parity, exposure definitions, and bot or employee filtering.
-
Verify pre-treatment balance and metric maturation windows.
-
Watch for peeking, multiple testing, novelty effects, interference, and instrumentation changes.
-
Distinguish intent-to-treat from treatment-on-treated analysis.
Part 3 - Metrics for a New Horizontal Home-Feed Module
Choose primary, secondary, and guardrail metrics for a new module in the homepage feed.
What This Part Should Cover
-
Define module-specific engagement such as impressions, CTR, saves, hides, and downstream actions.
-
Include user-level feed metrics such as sessions, DAU, retention, time quality, and content consumption.
-
Include guardrails for feed displacement, creator or merchant ecosystem health, latency, and revenue.
-
Avoid optimizing only per-exposed users when the decision is a user-level rollout.
Part 4 - Diagnose CTR Drop with DAU and Time Spent Up
Post-launch, homepage CTR drops in treatment while DAU and time spent increase. Explain how you would interpret and investigate this.
What This Part Should Cover
-
Check whether CTR denominator changed because impressions, ranking, or module mix changed.
-
Segment by user cohort, surface, device, geography, traffic source, and exposure intensity.
-
Examine click quality, downstream outcomes, retention, dwell time, hides, and long-click behavior.
-
Decide whether the CTR decline is harmful, a metric-definition artifact, or an acceptable trade-off.
Follow-up Questions
-
What would make you refuse to use the no-control analysis for launch?
-
How would you explain residual confounding to a product leader?
-
How would you design the corrected experiment after discovering the intern's mistake?