Find the Flaws in an End-to-End ML Pipeline Case Study Report

In this round you are given a written case study of a machine learning project and asked to point out everything that is wrong with it. The report covers the whole pipeline, is long, and includes a lot of tedious detail about data normalization. The condensed excerpt below is an illustrative stand-in written for practice, not the interview's actual report. It keeps the same shape: its sections run from the goal and the labels through deployment, and it includes detailed normalization steps. > **Goal.** Predict whether a user will make a purchase after being shown a promotional message, so the team can decide whom to show it to. > > 1. **Data and labels.** We exported all interaction logs available at export time. An interaction is labeled positive if the user made a purchase at any point between the interaction and the export date, and negative otherwise. > 2. **Cleaning.** Rows with a missing `age` were dropped. Duplicate interaction rows were removed after the train/test split, separately within each split. > 3. **Normalization.** Every numeric feature was standardized to zero mean and unit variance, using the mean and standard deviation of the full dataset. Remaining missing numeric values were then filled with 0. The skewed `recent_spend` feature was transformed with `log(x)` before standardization. The categorical `region_id` was mapped to integers and standardized like the other numeric features. > 4. **Features.** Besides profile features, we include `purchases_total`, the user's total number of purchases in the exported logs. > 5. **Class balance.** Positives are rare, so we duplicated positive rows until the classes were balanced, then split all rows randomly into train and test sets. > 6. **Model selection.** We trained several gradient-boosted tree models with different hyperparameters and report the one with the best test-set score. > 7. **Results.** The chosen model reaches very high accuracy on the test set, a large improvement over the previous rule-based targeting. > 8. **Deployment.** At serving time, each incoming batch of requests is standardized with that batch's own mean and standard deviation. The model will be retrained once a year. Identify the problems in this report. For each one, explain why it matters and how you would fix it, then rank the problems by how badly they undermine the report's conclusions. ```hint Follow the information in time order For every quantity the pipeline computes, ask which rows and which time period it was computed from, and whether that information would exist at the moment a prediction is made. ``` ```hint Not every flaw is equal Some problems only waste effort. Others make the headline result meaningless. Decide which is which before you start listing fixes. ``` ### Constraints and Clarifications - Treat the excerpt as the complete report: any step it does not mention was not done. - Every section of the report is open to review, from its stated goal to its deployment plan. ### Clarifying Questions - Who will read the review: the team that has to fix the pipeline, or the people deciding whether to deploy the model? - How will the model's scores be turned into the decision about whom to show the message to? - Is the project's data still available, so that a proposed fix can include rerunning part of the pipeline, or must the review work from the text alone? ### What a Strong Answer Covers - Coverage of every section of the report, from the stated goal through deployment - For each problem found, the mechanism by which it distorts the result or breaks the model, and a concrete fix - Reasoning about which data each step was computed from, and whether that data would exist when a prediction is made - Whether the reported evaluation supports the report's conclusion, and what evidence would support it - A prioritization that separates problems that invalidate the headline result from those that only weaken the model ### Follow-up Questions - Stakeholders have already seen the report's headline number. How do you present your findings to them, and how do you decide whether the project is still worth pursuing? - How would you validate a corrected model before it replaces the current rule-based targeting? - If you could ask the report's authors for only three more pieces of information, what would you ask for, and why? - Which of the problems you found could an automated check in the training pipeline catch, and how would each check work?