Build a robust ML pipeline

Read the full interview experience this question came from →

Quick Overview

This question evaluates a candidate's competency in designing robust end-to-end ML pipelines, covering temporal data slicing, leakage controls, time-series cross-validation, feature store consistency, offline and business metrics, drift detection and monitoring, online rollout strategies, retraining triggers, and fairness assessment.

Build a robust ML pipeline

Company: Flatiron Health

Role: Data Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Technical Screen

You inherit an ML pipeline that predicts next-7-day churn for users, but data quality is inconsistent and feature drift is suspected. A) Propose an end-to-end pipeline design covering: temporal data slicing (label window vs feature window), leakage controls (e.g., using only information available up to prediction time), cross-validation scheme appropriate for time series, and a feature store strategy that guarantees training/serving consistency. B) Define offline metrics (e.g., AUC, PR-AUC, calibration error) and business metrics (e.g., uplift in retention from targeted interventions). Specify how you would threshold scores to optimize a cost-sensitive objective with asymmetric costs. C) Describe concrete data quality and drift monitors: missingness rates, schema checks, training-serving skew, and feature drift using PSI/JS divergence with alert thresholds (e.g., PSI > 0.25 severe). Include how to separate drift in covariates from drift in the target due to product changes. D) Detail an online rollout plan: canary scoring, shadow mode, real-time monitoring, rollback triggers, and retraining cadence. Define explicit retraining triggers (e.g., weekly if PSI moderate for two consecutive weeks or business KPI degrades by X%). Address fairness checks across at least two sensitive cohorts and how you would mitigate disparities.

Overview: This question evaluates a candidate's competency in designing robust end-to-end ML pipelines, covering temporal data slicing, leakage controls, time-series cross-validation, feature store consistency, offline and business metrics, drift detection and monitoring, online rollout strategies, retraining triggers, and fairness assessment.

Read the full Flatiron Health Data Scientist interview experience this question came from

|Home/Machine Learning/Flatiron Health
Flatiron Health logo
Flatiron Health
Oct 13, 2025
mediumData ScientistTechnical ScreenMachine Learning
12
0

You inherit an ML pipeline that predicts next-7-day churn for users, but data quality is inconsistent and feature drift is suspected. A) Propose an end-to-end pipeline design covering: temporal data slicing (label window vs feature window), leakage controls (e.g., using only information available up to prediction time), cross-validation scheme appropriate for time series, and a feature store strategy that guarantees training/serving consistency. B) Define offline metrics (e.g., AUC, PR-AUC, calibration error) and business metrics (e.g., uplift in retention from targeted interventions). Specify how you would threshold scores to optimize a cost-sensitive objective with asymmetric costs. C) Describe concrete data quality and drift monitors: missingness rates, schema checks, training-serving skew, and feature drift using PSI/JS divergence with alert thresholds (e.g., PSI > 0.25 severe). Include how to separate drift in covariates from drift in the target due to product changes. D) Detail an online rollout plan: canary scoring, shadow mode, real-time monitoring, rollback triggers, and retraining cadence. Define explicit retraining triggers (e.g., weekly if PSI moderate for two consecutive weeks or business KPI degrades by X%). Address fairness checks across at least two sensitive cohorts and how you would mitigate disparities.

Loading comments...