How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a medium difficulty Machine Learning question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Evaluate and Experiment with Harmful Content Detection Model

Quick Overview

This question evaluates a candidate's competence in machine learning model evaluation and online experiment design for content moderation, testing skills such as handling class imbalance, probabilistic scoring and calibration, threshold selection, slice-based robustness checks, and production experiment planning.

Evaluating a Harmful-Content Detection Model: Offline and Online

Context

You are given a binary classification model that detects harmful content in a social platform and flags items for either removal or down‑ranking. You need to:

Evaluate the model offline on a labeled validation set.
Design an online experiment to test the model in production.

Assume class imbalance (harmful content is rare), probabilistic model outputs (scores), and that some actions (auto‑remove) can prevent us from observing true labels unless we design around it.

Tasks

Offline evaluation (labeled validation set):
- Define and compute core metrics (precision, recall, FPR, ROC/PR curves, AUCs).
- Assess calibration and choose an operating threshold given policy and cost trade‑offs.
- Check robustness across slices (e.g., language/region) and over time.
Online experiment design:
- State hypotheses.
- Define variants (control vs. treatment), including any shadow/canary ramps.
- Specify randomization unit, traffic split, duration, and significance plan.
- Define primary success metrics and guardrails (safety, engagement, fairness, latency).
- Address measurement challenges (delayed/hidden labels due to enforcement).

Quick Overview

Context

You are given a binary classification model that detects harmful content in a social platform and flags items for either removal or down‑ranking. You need to:

Evaluate the model offline on a labeled validation set.

Design an online experiment to test the model in production.

Assume class imbalance (harmful content is rare), probabilistic model outputs (scores), and that some actions (auto‑remove) can prevent us from observing true labels unless we design around it.

Tasks

Offline evaluation (labeled validation set):

Define and compute core metrics (precision, recall, FPR, ROC/PR curves, AUCs).
Assess calibration and choose an operating threshold given policy and cost trade‑offs.
Check robustness across slices (e.g., language/region) and over time.

Online experiment design:

State hypotheses.
Define variants (control vs. treatment), including any shadow/canary ramps.
Specify randomization unit, traffic split, duration, and significance plan.
Define primary success metrics and guardrails (safety, engagement, fairness, latency).
Address measurement challenges (delayed/hidden labels due to enforcement).

Evaluate and Experiment with Harmful Content Detection Model

Quick Overview

Evaluating a Harmful-Content Detection Model: Offline and Online

Context

Tasks

Solution

Comments (0)

Evaluate and Experiment with Harmful Content Detection Model

Quick Overview

Evaluating a Harmful-Content Detection Model: Offline and Online

Context

Tasks

Solution

Comments (0)