Causally measure traffic reduction effectiveness
Company: TikTok
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Traffic throttling () for flagged sellers was shipped without a clean A/B test. Measure its causal impact on abuse and business outcomes. (a) Specify the estimand (e.g., average treatment effect on treated sellers for complaint_per_1k_orders and GMV). (b) Propose two credible quasi-experimental designs and when each is valid: (1) Regression discontinuity around the risk-score threshold that triggers throttling; detail bandwidth selection, manipulation checks, and local polynomial specification. (2) Difference-in-differences/event study with staggered adoption; describe constructing a matched control group, testing pre-trends, choosing an estimator robust to staggered timing (e.g., Sun-Abraham), clustering of standard errors, and handling dynamic effects. (c) Define the analysis window, unit of observation, and how to handle treatment intensity (dose = percent rank penalty or impression share reduced). (d) List at least three robustness/validity checks: placebo thresholds, negative-control outcomes, falsification on pre-policy periods, sensitivity to trimming top-GMV sellers, and bounding bias from misclassification. (e) How would you detect and adjust for spillovers (e.g., seller migration to new accounts) and survivorship bias?
Quick Answer: This question evaluates skills in causal inference and quasi-experimental design, including specifying estimands, choosing RD and staggered DiD approaches, accounting for treatment intensity, and applying robustness checks and spillover detection when measuring policy impacts on abuse and business KPIs.