Tune fraud threshold under review capacity and costs

Q: Tune fraud threshold under review capacity and costs

This question evaluates a data scientist's competency in cost-sensitive threshold tuning, calibrated probability modeling, and operational decision-making for fraud triage, including formulating expected costs under review capacity and monetary trade-offs.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Machine Learning question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Fraud Triage Thresholding with Calibrated Scores

Context

You have a fraud model that outputs a calibrated score s ∈ [0, 1] per account, where s ≈ P(fake | features). Each day you must triage 2,000,000 accounts into one of three actions:

Auto-block if s ≥ t.
Send to manual review if r ≤ s < t.
Allow otherwise (s < r).

Constraints and costs:

Daily volume: 2,000,000 accounts.
Base fake rate: ~1% (for reference; calibration handles this implicitly).
Manual review budget: ≤ 100,000 accounts/day.
Manual review detects 95% of fakes it sees (and misses 5%).
Cost of auto-blocking a real user: $5.
Cost of letting a fake pass: $20.
Cost of manual review: $1 per account.

Assumption: Manual review does not incorrectly block real users (false blocks via review are negligible compared to auto-block false positives). If this is not true in your system, add that cost explicitly.

Tasks

(a) Formulate the expected daily cost as a function of thresholds r and t given calibrated scores. Describe how to estimate it from historical labeled data using isotonic or Platt calibration and empirical score distributions.

(b) Optimize r and t under the review budget. Explain how you would choose the operating point on the precision–recall (PR) curve and verify the choice with an online interleaved test.

(c) Describe a drift monitoring plan and a weekly threshold re-tuning process with backtesting and safety rails.

Tune fraud threshold under review capacity and costs

Quick Overview

Tune fraud threshold under review capacity and costs

Fraud Triage Thresholding with Calibrated Scores

Context

Tasks

Write your answer