How would you measure misinformation impact and recommendation bias?

You are interviewing for an experienced Data Scientist role on a short-form video platform (e.g., TikTok). Product sense / case questions come up frequently.

Case A — Estimate “bad AIGC / fake news” impact in 1 day

The platform has limited content reviewers. Leadership asks you to use one day to measure the impact of fake news / low-quality AIGC (“bad AIGC”) on the platform.

Task: Propose an end-to-end plan to estimate (1) how much “bad AIGC/fake news” exists and (2) its user impact, under tight time and labeling constraints.

Include:

• A working definition of “bad AIGC/fake news” and how you operationalize it.
• What metrics you would report (primary + diagnostics + guardrails).
• How you would sample content and users to get an estimate quickly.
• How you would quantify uncertainty (e.g., confidence intervals) and known limitations.
• How you would avoid common biases (selection, survivorship, reviewer inconsistency).

Case B — Validate claim of confirmation bias for minor users

Mainstream media claims there is confirmation bias in the recommendation system for minor (underage) users.

Task: Design an analysis/experiment to validate or refute the claim.

Include:

• How you define “confirmation bias” in a recommender context (measurable).
• What comparison baselines/counterfactuals you would use.
• How you would handle confounding (e.g., user preference vs algorithm effect), selection bias, and Simpson’s paradox.
• What slices you would analyze (age bands, geography, onboarding stage, etc.).
• What actions you would recommend depending on outcomes (mitigations, monitoring).