Diagnose and fix underperforming ML model

Quick Overview

This question evaluates a data scientist's competency in diagnosing and remediating underperforming binary classifiers under severe class imbalance, covering validation diagnostics, calibration, threshold selection under operational review constraints, cost-sensitive utility reasoning, and basic deployment monitoring for drift.

Diagnose and fix underperforming ML model

Company: Amazon

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

You inherited a binary fraud model with extreme class imbalance (positives ≈2%). Current performance on a temporally separated validation set: AUC=0.61, precision@recall=0.90 is only 0.05. You have one day to meaningfully improve recall at fixed review capacity. 1) Describe how you would quickly diagnose underfitting vs. overfitting (learning curves, calibration plots, PR vs. ROC trade-offs, leakage checks). 2) Propose three targeted interventions that can be implemented in a day (e.g., class-weighted loss, monotonic gradient boosting with categorical encoders, threshold moving with cost-sensitive utility) and justify why each should help. 3) Show how you would choose a decision threshold that maximizes expected utility given: FP cost=$2, FN cost=$50, review capacity=0.5% of traffic; write the utility formula and outline the validation-time procedure. 4) List the minimal logging/monitoring you’d add at deployment to detect drift and data quality issues within a week.

Quick Answer: This question evaluates a data scientist's competency in diagnosing and remediating underperforming binary classifiers under severe class imbalance, covering validation diagnostics, calibration, threshold selection under operational review constraints, cost-sensitive utility reasoning, and basic deployment monitoring for drift.

|Home/Machine Learning/Amazon
Amazon logo
Amazon
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenMachine Learning
8
0

Rapidly Improving Recall Under Class Imbalance (One-Day Plan)

Context

You inherit a binary fraud detection model with severe class imbalance (positive rate ≈ 2%). Evaluation on a temporally separated validation set shows:

  • ROC AUC = 0.61
  • Precision at 90% recall = 0.05 (very low precision at high recall, consistent with extreme imbalance)
  • Operations constraint: only 0.5% of traffic can be reviewed (fixed review capacity)

Goal: In one day, meaningfully improve recall at the same review capacity.

Tasks

  1. Diagnosis: Describe how you would quickly distinguish underfitting versus overfitting using learning curves, calibration plots, PR vs ROC analysis at fixed capacity, and leakage/drift checks.
  2. Interventions: Propose three changes you can implement in a day (e.g., class-weighted loss, monotonic gradient boosting with categorical encoders, threshold moving using cost-sensitive utility), and justify why each helps.
  3. Thresholding for Utility: Show how to choose a decision threshold that maximizes expected utility given:
  • False Positive (FP) cost = $2
  • False Negative (FN) cost = $50
  • Review capacity = 0.5% of traffic Provide the utility (or cost) formula and outline the selection procedure on validation data.
  1. Monitoring: List the minimal logging/monitoring to add at deployment to detect drift and data quality issues within a week.
Loading comments...