Plan How Raw Text Becomes LLM Training Data: Filtering, Dedup, Mixing, Evaluation

Walk through how you would turn raw text into a curated training set for a large language model, from choosing sources to the data the training job finally reads. ```hint Judge filters by the model Every filtering or deduplication choice removes tokens. Think about how you would find out whether a given step made the trained model better, not just the data cleaner. ``` ### Clarifying Questions - Is this for pretraining a base model from scratch, or for fine-tuning data such as instruction or preference examples, where volumes and quality checks differ? - Am I curating data for a model my own team trains, or designing a curation product that NVIDIA's customers run on their own data? - What capabilities, languages and domains should the model have (general chat, code, a specific industry), and roughly what token budget does the training plan allow? - Are there licensing, privacy or regional restrictions on which sources can be used? ### What a Strong Answer Covers - The stages from sourcing to training-ready shards: extraction, language identification, quality filtering, deduplication, safety and personal-data handling, benchmark decontamination, and mixing. - How quality is defined and measured, including classifier-based filtering and the bias it can introduce. - How curation choices are validated with small proxy training runs and held-out evaluations. - Scale, cost and reproducibility: throughput on very large corpora, GPU acceleration, versioning and lineage. - The trade-off between quality and quantity, plus governance concerns such as licensing and opt-outs. ### Follow-up Questions - After aggressive filtering you have too few high-quality tokens for the planned training run. What are your options, and what does each cost? - A model trained on the new dataset scores higher on public benchmarks, but users say it got worse in a language you care about. How do you investigate? - If you were the PM for a GPU-accelerated curation library, which stage would you accelerate first, and how would you measure adoption?