Investigate Homepage Experiment Without Control Group: Methods and Metrics
Company: Pinterest
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Several experimentation and product-metric challenges for a social-media homepage.
##### Question
An intern launched an experiment without a control group—how can you still estimate treatment impact? Compare matching versus propensity-score weighting and discuss their trade-offs. Review an existing A/B test and list common pitfalls that could bias the results. For a new horizontal home-feed module, what primary metrics would you track to judge success? Post-launch you observe homepage click-through dropping in treatment while DAU and time-spent stay flat—how would you investigate root causes and which user segments would you analyze first?
##### Hints
Think causal inference, experiment design, diagnosable metrics hierarchy, segmentation by device, geography, tenure, power-users vs casual, etc.
Quick Answer: Evaluates quasi-experimental analysis and product-metric judgment after a homepage launch lacks a randomized control. Strong answers compare matching and propensity weighting, audit experiment hygiene, choose metrics, and diagnose mixed CTR results.
A social-media homepage team is analyzing a personalized feed. An intern accidentally launched a treatment to a user cohort without a randomized control group. You have event logs, impression and click events, pre-period user behavior, device, geo, app version, and eligibility flags.
Answer the tasks below as if you are advising whether the results can still support a product decision.
Constraints & Assumptions
Acknowledge that causal certainty is weaker without randomized control.
Use pre-treatment data only for matching, weighting, and diagnostics.
Check overlap, balance, logging quality, and maturation before interpreting treatment effects.
Separate exploratory directional evidence from a launch-grade causal estimate.
Clarifying Questions to Ask Guidance
Why did the cohort receive treatment, and were untreated users eligible under the same rules?
Is there a clean pre-period and post-period for every user?
Which metric is the decision metric: CTR, retention, DAU, time spent, module engagement, or revenue?
Can we rerun the experiment with a proper randomized holdout?
Part 1 - Estimate Impact Without a Randomized Control
Compare matching and propensity-score weighting for estimating treatment impact.
What This Part Should Cover Guidance
Define treated and candidate comparison users using eligibility and pre-period covariates.
Explain exact matching, nearest-neighbor matching, or coarsened matching and when each is useful.
Explain propensity-score weighting, overlap, trimming, and balance diagnostics.
State assumptions such as conditional ignorability, positivity, and no interference.
Recommend sensitivity checks and, if possible, a future randomized experiment.
Part 2 - A/B Test Hygiene Review
List common randomized experiment pitfalls that could bias an existing A/B test.
What This Part Should Cover Guidance
Check sample ratio mismatch, assignment stickiness, logging parity, exposure definitions, and bot or employee filtering.
Verify pre-treatment balance and metric maturation windows.
Watch for peeking, multiple testing, novelty effects, interference, and instrumentation changes.
Distinguish intent-to-treat from treatment-on-treated analysis.
Part 3 - Metrics for a New Horizontal Home-Feed Module
Choose primary, secondary, and guardrail metrics for a new module in the homepage feed.
What This Part Should Cover Guidance
Define module-specific engagement such as impressions, CTR, saves, hides, and downstream actions.
Include user-level feed metrics such as sessions, DAU, retention, time quality, and content consumption.
Include guardrails for feed displacement, creator or merchant ecosystem health, latency, and revenue.
Avoid optimizing only per-exposed users when the decision is a user-level rollout.
Part 4 - Diagnose CTR Drop with DAU and Time Spent Up
Post-launch, homepage CTR drops in treatment while DAU and time spent increase. Explain how you would interpret and investigate this.
What This Part Should Cover Guidance
Check whether CTR denominator changed because impressions, ranking, or module mix changed.
Segment by user cohort, surface, device, geography, traffic source, and exposure intensity.
Examine click quality, downstream outcomes, retention, dwell time, hides, and long-click behavior.
Decide whether the CTR decline is harmful, a metric-definition artifact, or an acceptable trade-off.
Follow-up Questions Guidance
What would make you refuse to use the no-control analysis for launch?
How would you explain residual confounding to a product leader?
How would you design the corrected experiment after discovering the intern's mistake?