Address Fraud Detection with Imbalance and Concept Drift Solutions

Quick Overview

Evaluates end-to-end fraud detection ML system design for online payments under drift and imbalance. Strong answers handle delayed labels, class imbalance, feature engineering, cost-sensitive validation, real-time deployment, threshold actions, monitoring, retraining, and concept drift.

Address Fraud Detection with Imbalance and Concept Drift Solutions

Company: Netflix

Role: Data Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Onsite

##### Scenario You are tasked with building a fraud-detection model for an online payments product. ##### Question Outline the end-to-end ML workflow: data collection, feature engineering, model selection, validation, deployment, and monitoring. How would you handle severe class imbalance and concept drift in this context? ##### Hints Discuss resampling, cost-sensitive learning, ROC-AUC, sliding windows, and automated retraining triggers.

Overview: Evaluates end-to-end fraud detection ML system design for online payments under drift and imbalance. Strong answers handle delayed labels, class imbalance, feature engineering, cost-sensitive validation, real-time deployment, threshold actions, monitoring, retraining, and concept drift.

|Home/Machine Learning/Netflix
Netflix logo
Netflix
Jul 12, 2025
mediumData ScientistOnsiteMachine Learning
17
0

Address Fraud Detection with Imbalance and Concept Drift Solutions

You are building a fraud-detection model for an online payments product that must score transactions in near real time. Fraud is rare, labels arrive with delay, and fraud patterns evolve over time.

Constraints & Assumptions

  • Treat this as an end-to-end ML system design question.
  • Include business actions such as approve, review, challenge, or decline.
  • Handle class imbalance and concept drift explicitly.
  • Include deployment and monitoring, not only offline modeling.

Clarifying Questions to Ask Guidance

  • What is the fraud label source and how delayed is it?
  • What action will the model trigger at each risk threshold?
  • What are the relative costs of false positives and false negatives?
  • What latency and reliability requirements exist?

Part 1 - Data and Labels

How would you collect data and define labels?

What This Part Should Cover Guidance

  • Transaction, user, device, merchant, IP, geolocation, graph, historical behavior, chargeback, review, and third-party risk data.
  • Delayed labels, noisy labels, leakage prevention, and train/serve consistency.

Part 2 - Features and Models

How would you engineer features and select models?

What This Part Should Cover Guidance

  • Real-time aggregates, velocity features, graph features, device fingerprints, merchant risk, user history, and anomaly signals.
  • Baselines, logistic regression, gradient boosting, deep or graph models, rules plus ML, calibration, and latency constraints.

Part 3 - Validation and Imbalance

How would you validate the model and handle severe class imbalance?

What This Part Should Cover Guidance

  • Time-based splits, PR-AUC, ROC-AUC, precision at review capacity, recall, cost-based metrics, calibration, and thresholding.
  • Resampling, class weights, focal/cost-sensitive loss, and business-cost optimization.

Part 4 - Deployment, Monitoring, and Drift

How would you deploy, monitor, and retrain the system under concept drift?

What This Part Should Cover Guidance

  • Shadow mode, canary, monitoring approval rate, fraud rate, false declines, chargebacks, appeals, drift, data quality, and latency.
  • Sliding windows, automated retraining triggers, champion/challenger, and human review feedback loops.

What a Strong Answer Covers Guidance

A strong answer treats fraud detection as an operational risk system, balancing fraud loss, customer friction, label delay, imbalance, adversarial drift, and real-time deployment constraints.

Follow-up Questions Guidance

  • How would you choose thresholds for auto-decline versus manual review?
  • What if fraud labels arrive 60 days later?
  • How would you detect a new fraud attack pattern?
Loading comments...

Browse More Questions

More Machine Learning•More Netflix•More Data Scientist•Netflix Data Scientist•Netflix Machine Learning•Data Scientist Machine Learning