How do I approach ML System Design interview questions?

ML System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master ml system design interviews.

What difficulty level is this interview question?

This is a hard difficulty ML System Design question, commonly asked during Onsite rounds at PayPal.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at PayPal during technical interviews.

Design fraud detection from raw transactions

Quick Overview

This question evaluates a candidate's ability to design an end-to-end transaction fraud detection system, testing competencies in feature engineering, handling delayed and partial labels and class imbalance, real-time feature retrieval and model serving, decision thresholding under manual-review constraints, feedback-loop integration, monitoring, and backtesting. It is commonly asked in ML system design interviews to assess both conceptual understanding and practical application of production machine learning and data engineering trade-offs in the ML System Design domain for data scientist roles.

System Design: End-to-End Transaction Fraud Detection

Context

You are given a large, multi-table dataset of transactions and customer/merchant metadata. Fraud labels arrive with delays (e.g., chargebacks weeks later) and may be partial (e.g., only for reviewed or disputed transactions). Design an end-to-end system to decide, in real time, whether to approve, decline, or send each transaction to manual review.

Requirements

Cover the following aspects with clear assumptions and rationale:

Data and Feature Engineering
- Velocity features (multi-horizon counts/sums/uniques).
- Graph/link features across entities (user/card/email/device/IP/merchant).
- Device and IP signals (fingerprinting, geolocation, proxy/TOR, ASN risk).
Labels, Class Imbalance, and Latency
- Handling severe class imbalance.
- Handling delayed/partial labels and selective-label bias.
Training and Validation Splits
- Splitting to avoid leakage in time and across entities.
- Ensuring offline/online feature parity.
Decision Thresholding and Review Capacity
- Approve/Decline/Review policy with cost-sensitive thresholds.
- Meeting a fixed manual review capacity.
Real-Time Scoring and Latency Budgets
- Online feature retrieval and model serving under strict latency.
- Fallbacks and degradation strategies.
Feedback Loops
- Incorporating manual review outcomes and chargebacks.
- Exploration/holdout strategies to mitigate bias.
Monitoring and Alerting
- Drift (input/output), TPR/FPR with delayed labels, calibration, approval/decline rates, review queue health.
Backtesting Plan
- Time-ordered replay, off-policy evaluation, simulation of review capacity, metrics and confidence intervals.

State reasonable assumptions if needed and justify key design choices.

Quick Overview

Context

Requirements

Cover the following aspects with clear assumptions and rationale:

Data and Feature Engineering

Velocity features (multi-horizon counts/sums/uniques).
Graph/link features across entities (user/card/email/device/IP/merchant).
Device and IP signals (fingerprinting, geolocation, proxy/TOR, ASN risk).

Labels, Class Imbalance, and Latency

Handling severe class imbalance.
Handling delayed/partial labels and selective-label bias.

Training and Validation Splits

Splitting to avoid leakage in time and across entities.
Ensuring offline/online feature parity.

Decision Thresholding and Review Capacity

Approve/Decline/Review policy with cost-sensitive thresholds.
Meeting a fixed manual review capacity.

Real-Time Scoring and Latency Budgets

Online feature retrieval and model serving under strict latency.
Fallbacks and degradation strategies.

Feedback Loops

Incorporating manual review outcomes and chargebacks.
Exploration/holdout strategies to mitigate bias.

Monitoring and Alerting

Drift (input/output), TPR/FPR with delayed labels, calibration, approval/decline rates, review queue health.

Backtesting Plan

Time-ordered replay, off-policy evaluation, simulation of review capacity, metrics and confidence intervals.

State reasonable assumptions if needed and justify key design choices.

Design fraud detection from raw transactions

Quick Overview

System Design: End-to-End Transaction Fraud Detection

Context

Requirements

Solution

Comments (0)

Design fraud detection from raw transactions

Quick Overview

System Design: End-to-End Transaction Fraud Detection

Context

Requirements

Solution

Comments (0)