Messenger, Group Calling, And Networked Adoption

What's being tested

These prompts probe a Data Scientist's ability to design and analyze experiments and observational analyses where network effects / interference matter: picking the right unit of randomization, defining an Overall Evaluation Criterion (OEC) that captures primary and spillover value, and using statistical methods robust to correlated outcomes (clustered or networked). Interviewers want to see causal reasoning (how to estimate direct vs indirect effects), pragmatic power/sample-size calculations with ICC or design effect adjustments, and concrete analysis plans (pre-specification, guardrails, monitoring).

Core knowledge

• Interference / spillovers: interference violates SUTVA; treatment for one unit can change others' outcomes. Distinguish direct, indirect, and total effects (Hudgens & Halloran-style framing) when writing hypotheses.

• Unit of randomization choices: individual randomization (max power, high contamination risk) vs cluster randomization (reduced contamination, lower effective N). Decide using network topology and cross-cluster edge volume.

• Design effect & ICC: adjust sample size for clustering using design effect: $DE = 1 + (m-1) \cdot ICC$ where $m$ is average cluster size. Effective sample size ≈ $N/DE$.

• Graph cluster randomization: partition users with community detection (e.g., Louvain, METIS) or edge-cut minimization to keep most edges intra-cluster; balance cluster sizes to control power loss.

• Exposure models: summarize a user's neighborhood treatment as features (e.g., fraction of friends treated, count of treated neighbors, time-weighted exposure) and pre-define thresholds or continuous models for indirect-effect estimation.

• Randomization inference / permutation tests: with interference, use randomization-based p-values (shuffle cluster assignments consistent with design) to get exact-type I error control when asymptotics fail.

• Metric design (OEC and guardrails): pick a single OEC (e.g., weekly unique group-call participants per DAU) plus guardrails (call quality p99 drop, spam flags, support tickets, ARPU). Pre-specify secondary metrics and multiplicity control (Benjamini–Hochberg or hierarchical testing).

• Power for count/rare events: use Poisson or negative-binomial models and simulate power when base rates are low; analytic approximations fail for sparse outcomes or highly skewed cluster sizes.

• Observational log analysis patterns: dedupe sessions with ROW_NUMBER() OVER (PARTITION BY user ORDER BY ts DESC) for last event, sessionize with idle-time thresholds, and build pair-unique interactions using canonical (min,max) ID tuples.

• Causal identification strategies: if randomization infeasible, use instrumental variables (encouragement designs), difference-in-differences with parallel-trends checks, and regression with network-fixed effects or cluster-robust SEs.

• Monitoring and stopping rules: pre-specify interim checks for guardrails, use alpha-spending or sequential testing frameworks (e.g., O’Brien–Fleming), and avoid peeking on OEC without adjusted thresholds.

Worked example — Design and analyze a group-calls experiment

First 30s: ask what success looks like (engagement, retention, revenue), constraints (can we randomize at cluster level?), and expected network structure (average degree, existence of tight friend groups). Skeleton answer pillars: (1) randomization unit and clustering algorithm (graph-cluster randomization using Louvain to keep edges internal), (2) OEC and guardrails (weekly unique call participants per DAU; guardrails: call drops, spam rate, support tickets), (3) analysis plan accounting for interference (estimate direct vs indirect effects via exposure models; use randomization inference for p-values), (4) power and monitoring (simulate with observed degree distribution; compute DE using ICC estimate). Key tradeoff to flag: larger clusters reduce contamination but inflate design effect and reduce power — quantify by simulating effective sample size. Close with next steps: run a pilot A/A to validate logging and ICC, simulate treatment diffusion, and pre-register analysis plan with multiple-hypothesis controls.

A second angle — Analyze Recent User Activity from Video Call Logs

This is observational aggregation rather than randomized inference, but the same network concerns appear as features. First clarify retention windows (7/28/90-day), canonicalize IDs (account vs device), and define what counts as an interaction (joined > X seconds). Skeleton: (1) SQL dedupe using ROW_NUMBER() to get last activity per user; (2) compute unique pair interactions with canonical (LEAST(user1,user2), GREATEST(...)); (3) generate exposure covariates (num unique contacts in last 7 days) to use as predictors in downstream models. Differences: causal claims are limited without exogenous variation — use quasi-experimental methods (instrumental variables, regression discontinuity if available) or treat outputs as diagnostics for experiment design (estimate ICC, degree distributions, baseline rates) to power future randomized tests.

Common pitfalls

Pitfall: Treating users as independent in analysis — ignoring clustering/interference underestimates variance and will inflate false positives. Always compute cluster-robust SEs or use randomization inference.

Pitfall: Vagueness about the OEC — proposing many primary metrics or leaving guardrails unspecified leads to post-hoc selection. Pre-specify one OEC, ranked secondaries, and multiplicity rules.

Pitfall: Over-relying on analytic power formulas without simulation — skewed cluster-size distributions, low base rates, or overdispersion break closed-form approximations; simulate using the real network and event-rate assumptions.

Connections

Interviewers often pivot to adjacent topics: A/B testing for spam / safety where clustering preserves attacker behavior, or monetization experiments (ARPU uplift from in-call purchases) which layer revenue attribution on the same network-aware framework. They may also move toward recommendation/ranking impacts on call invitations and downstream engagement.

Further reading

• Hudgens, M. G. & Halloran, M. E. — "Toward causal inference with interference" — formal definitions of direct/indirect effects and randomized designs under interference.

• Ugander et al. — "Graph cluster randomization" (conference/paper) — practical methods to partition networks for cluster-level experiments and reduce bias from spillovers.