Estimate unbiased ad scores with many reviewers
Company: Google
Role: Data Scientist
Category: Statistics & Math
Difficulty: Medium
Interview Round: Technical Screen
You have 1,000 ads and 100 reviewers; each reviewer rates 100 ads on a 1–10 scale with incomplete overlap. Specify a mixed-effects model to estimate latent ad quality while debiasing reviewer severity and scale, e.g., yij = μ + qi (ad effect) + bj (reviewer intercept) + εij, with optional reviewer-specific scale or slope. State identifiability constraints, justify which effects are fixed vs random, and describe how to obtain BLUPs or Bayesian posteriors with uncertainty. Address heteroskedasticity, missingness, and rater drift. Propose validation (leave-one-reviewer-out, posterior predictive checks, ICC) and a principled way to rank ads with uncertainty-aware intervals.
Quick Answer: This question evaluates a candidate's skills in hierarchical and mixed-effects modeling, latent-variable estimation, debiasing rater severity and scale, model identifiability, and uncertainty quantification including BLUPs or Bayesian posterior inference.