Design metrics for violating content exposure

Q: Design metrics for violating content exposure

This question evaluates competency in metric design and measurement for content safety, including defining exposure metrics, inclusion/exclusion rules, handling label latency and appeals, uncertainty-aware alerting, and A/B test design for a Data Scientist role in the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Measuring User Exposure to Violating Content on a UGC Platform

Context

You work on a large-scale user-generated content (UGC) platform that uses automated and human moderation. You need a robust metric framework to quantify and monitor how much violating content users are exposed to, and to evaluate interventions that reduce this exposure.

Assumptions (define or adapt as needed):

A "view" is a content render that meets viewability thresholds (e.g., visible ≥1s, ≥50% viewport).
A "session" is a sequence of user activity separated by ≥30 minutes of inactivity.
DAU is the count of distinct real human users with ≥1 session in the day.
A "violating item" is a post/video/comment that violates policy after review (human or high-confidence auto) subject to appeals.

Tasks

Define precise formulas for at least three daily metrics and their 7-day rolling counterparts:
- view_prevalence = violating_views / total_views
- violating_session_rate = sessions_with_≥1_violating_view / total_sessions
- violations_per_active_user = violating_views / DAU
Specify exact inclusion/exclusion rules for what counts as a violating view under two timing regimes:
- Ex-ante: only violations known at the time of view
- Ex-post: final decision after reviews Include how to treat late-arriving labels, appeals, deleted content, repeat views, and bot traffic.
Is view_prevalence a good north-star metric? Compare it with:
- incident_rate = violating_items / items_created
- user_prevalence = users_exposed / DAU Discuss tradeoffs: detection lag, denominator gaming, precision/recall shifts, Simpson’s paradox across countries/surfaces, and Goodhart’s law.
Propose a weekly alerting method with thresholds using uncertainty estimates (e.g., Wilson or Bayesian beta–binomial intervals). Describe guardrail metrics (e.g., false positive exposure, creator churn, review queue SLA).
Sketch an A/B test to reduce view_prevalence: state the primary metric, key segments (country, surface, creator cohort), power assumptions, and how you’ll correct for label latency and selection bias when violations are discovered after exposure.

Design metrics for violating content exposure

Quick Overview

Measuring User Exposure to Violating Content on a UGC Platform

Context

Tasks

Solution

Comments (0)