Build a robust ML pipeline

Quick Overview

This question evaluates a candidate's competency in designing robust end-to-end ML pipelines, covering temporal data slicing, leakage controls, time-series cross-validation, feature store consistency, offline and business metrics, drift detection and monitoring, online rollout strategies, retraining triggers, and fairness assessment.

Build a robust ML pipeline

Company: Flatiron Health

Role: Data Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Technical Screen

You inherit an ML pipeline that predicts next-7-day churn for users, but data quality is inconsistent and feature drift is suspected. A) Propose an end-to-end pipeline design covering: temporal data slicing (label window vs feature window), leakage controls (e.g., using only information available up to prediction time), cross-validation scheme appropriate for time series, and a feature store strategy that guarantees training/serving consistency. B) Define offline metrics (e.g., AUC, PR-AUC, calibration error) and business metrics (e.g., uplift in retention from targeted interventions). Specify how you would threshold scores to optimize a cost-sensitive objective with asymmetric costs. C) Describe concrete data quality and drift monitors: missingness rates, schema checks, training-serving skew, and feature drift using PSI/JS divergence with alert thresholds (e.g., PSI > 0.25 severe). Include how to separate drift in covariates from drift in the target due to product changes. D) Detail an online rollout plan: canary scoring, shadow mode, real-time monitoring, rollback triggers, and retraining cadence. Define explicit retraining triggers (e.g., weekly if PSI moderate for two consecutive weeks or business KPI degrades by X%). Address fairness checks across at least two sensitive cohorts and how you would mitigate disparities.

Quick Answer: This question evaluates a candidate's competency in designing robust end-to-end ML pipelines, covering temporal data slicing, leakage controls, time-series cross-validation, feature store consistency, offline and business metrics, drift detection and monitoring, online rollout strategies, retraining triggers, and fairness assessment.

|Home/Machine Learning/Flatiron Health
Flatiron Health logo
Flatiron Health
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenMachine Learning
11
0

You inherit an ML pipeline that predicts next-7-day churn for users, but data quality is inconsistent and feature drift is suspected. A) Propose an end-to-end pipeline design covering: temporal data slicing (label window vs feature window), leakage controls (e.g., using only information available up to prediction time), cross-validation scheme appropriate for time series, and a feature store strategy that guarantees training/serving consistency. B) Define offline metrics (e.g., AUC, PR-AUC, calibration error) and business metrics (e.g., uplift in retention from targeted interventions). Specify how you would threshold scores to optimize a cost-sensitive objective with asymmetric costs. C) Describe concrete data quality and drift monitors: missingness rates, schema checks, training-serving skew, and feature drift using PSI/JS divergence with alert thresholds (e.g., PSI > 0.25 severe). Include how to separate drift in covariates from drift in the target due to product changes. D) Detail an online rollout plan: canary scoring, shadow mode, real-time monitoring, rollback triggers, and retraining cadence. Define explicit retraining triggers (e.g., weekly if PSI moderate for two consecutive weeks or business KPI degrades by X%). Address fairness checks across at least two sensitive cohorts and how you would mitigate disparities.

Loading comments...