Explain Data Collection Choices and Their Impact on Downstream Model Performance

In a short screening call for a machine learning engineering role, after brief introductions, the interviewer asks you to explain the data side of a project on your resume: how the data was collected and how it was processed. The conversation then turns to your understanding of how data affects a model's performance on downstream tasks. ### Clarifying Questions - Is the interviewer most interested in pre-training data, fine-tuning data, or evaluation data? - Which downstream tasks matter most to the team? - Should I choose the project with the most substantial data work, even if it is not the most recent one? ### Part 1 — Data collection and processing in your project Walk through how the data for one of your projects was collected and processed, from its sources to the form the model was trained on. ```hint Follow one record Trace a single raw record from its source to the form the model consumed, and name every step that changed it or could have removed it. ``` #### What This Part Should Cover - Sources, sampling and the reasons for choosing them - Cleaning, filtering, labeling and the construction of train, validation and test splits - Quality checks and protection against leakage between splits - Decisions backed by numbers rather than description alone ### Part 2 — How data affects downstream tasks Explain, with concrete mechanisms, how choices made in data collection and processing change a model's performance on downstream tasks, and how you would measure those effects rather than assert them. ```hint Mechanism, then measurement For each effect you name, say which experiment or measurement would show it. ``` #### What This Part Should Cover - Specific mechanisms such as coverage and distribution shift, label noise, duplication, contamination of evaluation data, and the mix of data sources - Experimental designs that isolate the effect of a data change - Trade-offs, such as quantity versus quality and filtering versus diversity ### What a Strong Answer Covers - First-hand detail from the candidate's own project rather than textbook generalities - Causal claims about data backed by experiments or measurements - Awareness of leakage and of evaluation data contaminating training data ### Follow-up Questions - A quality filter removed a large share of your data, and downstream accuracy dropped. How do you find out why? - How would you detect that a benchmark you report on has leaked into your training data? - How would you decide the mixing proportions between several data sources? - Your labels come from annotators who often disagree. What do you do?