Tune metrics for imbalanced classification

Quick Overview

This question evaluates a data scientist's competency in machine learning for rare-event detection, testing skills in preprocessing messy data, handling high-cardinality categoricals, designing validation splits, selecting imbalanced evaluation metrics and cost-sensitive decision thresholds, and reasoning about operational trade-offs.

Tune metrics for imbalanced classification

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Onsite

You must detect fraudulent transactions where positives are 0.5% of all cases. Data has 10% missing values, heavy outliers, and categorical high-cardinality features. (a) Propose a preprocessing pipeline for missing values and outliers. (b) Choose training/validation splits and justify stratified cross-validation. (c) Select evaluation metrics; compare ROC-AUC vs PR-AUC vs F1 at a chosen threshold; define cost-sensitive objective. (d) Give a concrete method to address class imbalance (e.g., calibrated class weights, focal loss, or SMOTE with care to avoid leakage). (e) Provide a real-world example where false positives are costlier than false negatives (or vice versa) and how that changes thresholding and monitoring.

Overview: This question evaluates a data scientist's competency in machine learning for rare-event detection, testing skills in preprocessing messy data, handling high-cardinality categoricals, designing validation splits, selecting imbalanced evaluation metrics and cost-sensitive decision thresholds, and reasoning about operational trade-offs.

|Home/Machine Learning
Oct 13, 2025
hardData ScientistOnsiteMachine Learning
4
0

Fraud Detection With Rare Positives (0.5%) and Messy Data

You are designing a supervised transaction-level fraud detector. Positives (fraud) are rare at 0.5% of all cases. The dataset has ~10% missing values, heavy-tailed outliers, and high-cardinality categorical features (e.g., merchant_id, device_id).

Answer the following:

  1. Preprocessing a) Propose a concrete preprocessing pipeline to handle missing values and outliers for both numeric and categorical features. Address high-cardinality categoricals and leakage prevention.
  2. Training/Validation Splits b) Specify how you would split the data for training/validation/testing. Justify stratified cross-validation (and when to prefer time-aware or group-aware schemes).
  3. Evaluation and Costing c) Select and justify evaluation metrics in this imbalanced setting. Compare ROC-AUC vs PR-AUC vs F1 at a chosen threshold. Define a cost-sensitive objective and the optimal decision threshold given costs.
  4. Class Imbalance d) Provide a concrete method to address class imbalance (e.g., calibrated class weights, focal loss, or SMOTE). Explain exactly how to apply it without leakage.
  5. Business Trade-offs e) Give a real-world example where false positives are costlier than false negatives (or vice versa), and explain how that changes thresholding and monitoring in production.
Loading comments...

Browse More Questions

More Machine Learning•More Data Scientist•Data Scientist Machine Learning