Evaluate friend-interaction feature with network interference

Read the full interview experience this question came from →

Quick Overview

This question evaluates a data scientist's competency in causal inference and network-aware experiment design, covering randomization under interference, exposure-weighted metric specification, power estimation with intracluster correlation, and cluster-robust analysis.

Evaluate friend-interaction feature with network interference

Company: Roblox

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: HR Screen

You plan to ship a “friend interaction boost” that ranks feed content higher when friends interact with it. Because interactions propagate through the social graph, standard user-level randomization violates SUTVA. Design an experiment that credibly estimates causal lift: 1) Randomization unit: Choose and defend graph-cluster randomization (e.g., Louvain clusters) vs ego-network clustering vs geo/time switchbacks. How will you quantify and cap cross-cluster edge cut ratio and exposure contamination? 2) Metrics: Define primary (session time, meaningful interactions) and guardrail metrics (spam reports, creator revenue cannibalization). Specify exposure-weighted metrics for partially treated users. 3) Power: Given average cluster size m and intracluster correlation ρ, derive effective sample size n_eff ≈ (K·m) / [1+(m−1)ρ]. Show how this changes required duration vs user-level AB. 4) Analysis: Outline cluster-robust variance estimation, CUPED with pre-period outcomes, and intent-to-treat vs exposure-on-treated estimands. How do you handle creators whose audiences span treatment/control clusters? 5) Diagnostics & fallbacks: Pre-commit spillover checks, negative controls, and a holdout of high-degree nodes. If contamination is too high mid-test, propose a redesign that preserves inference while limiting blast radius.

Overview: This question evaluates a data scientist's competency in causal inference and network-aware experiment design, covering randomization under interference, exposure-weighted metric specification, power estimation with intracluster correlation, and cluster-robust analysis.

Read the full Roblox Data Scientist interview experience this question came from

|Home/Analytics & Experimentation/Roblox
Roblox logo
Roblox
Oct 13, 2025
hardData ScientistHR ScreenAnalytics & Experimentation
9
0

Experiment Design: Network-Aware Test for a "Friend Interaction Boost" in Feed Ranking

Context

You plan to ship a ranking change that boosts items in the feed when a user's friends have interacted with them. Because interactions propagate through the social graph, the Stable Unit Treatment Value Assumption (SUTVA) is violated under user-level randomization: a user's outcome can be affected by neighbors' assignments. Design a credible experiment that estimates causal lift under network interference.

Tasks

  1. Randomization Unit
  • Choose and defend one of: graph-cluster randomization (e.g., Louvain or partitioning), ego-network clustering, or geo/time switchbacks.
  • Define how you will quantify and cap:
    • Cross-cluster edge cut ratio.
    • Exposure contamination across treatment arms.
  1. Metrics
  • Define primary metrics (e.g., session time, meaningful interactions) and guardrail metrics (e.g., spam reports, creator revenue cannibalization).
  • Specify exposure-weighted metrics for partially treated users.
  1. Power
  • Given average cluster size m and intracluster correlation ρ, derive the effective sample size: n_eff ≈ (K · m) / [1 + (m − 1)ρ] where K is the number of clusters.
  • Explain how this changes required test duration relative to a user-level A/B test.
  1. Analysis
  • Outline cluster-robust variance estimation; use CUPED with pre-period outcomes.
  • Define intent-to-treat (ITT) vs exposure-on-treated estimands and how to estimate each.
  • Describe how to handle creators whose audiences span both treatment and control clusters.
  1. Diagnostics and Fallbacks
  • Pre-commit spillover checks, negative controls, and a holdout of high-degree nodes.
  • If contamination is too high mid-test, propose a redesign that preserves inference while limiting blast radius.
Loading comments...