Single-Pass Python Cleaning Pipeline for Million-Document Web Crawl Parquet Files

You have 48 hours for a take-home assignment: build a data cleaning pipeline. The input is a set of Parquet files that hold on the order of a million documents crawled from public web pages. The pipeline must read the input in a single pass and write out the cleaned documents. The deliverable is one Python script. After you submit it, you walk a hiring manager through your solution in a 15-minute live session. ### Clarifying Questions - What is the input schema: which columns hold the URL, the extracted text, and any metadata such as crawl time or content type? - Which cleaning steps are required: exact or near-duplicate removal, language filtering, length or quality filters, boilerplate removal, removal of personal information? - What output format and layout are expected, and should dropped records be kept separately with the reason they were dropped? - Does "read once" forbid any second pass over the data, or only re-reading the raw files? - What machine will run it (memory, cores), and is there a time budget? - Which third-party libraries may the script use? ### Part 1 — Decide what "clean" means Define the cleaning steps your script applies to crawled web text, in order, and justify each one and its thresholds. ```hint Look before you filter Sample documents from the input and list the concrete kinds of noise you see before choosing any rule or threshold. ``` #### What This Part Should Cover - The difference between normalizing a document, dropping it, and removing duplicates - Rules and thresholds justified by what the data actually contains - What is kept or added as metadata, and what is discarded ### Part 2 — A single-pass, bounded-memory implementation Implement the pipeline as one Python script that reads each input record once and streams its output. ```hint What must be remembered Anything you need across records, such as which documents you have already seen, must stay in memory during the single pass. Estimate how large that state gets for this input size. ``` #### What This Part Should Cover - Streaming reads and writes instead of loading everything at once - The memory needed for state carried across records, estimated for the input size - Throughput, and where parallelism can help without breaking the single pass - Robustness: missing or malformed rows, schema mismatches, and a crash midway ### Part 3 — The 15-minute walk-through Plan what you will show and say to the hiring manager in 15 minutes. ```hint Evidence for each rule Plan to show, for each rule, how many documents it removed and a few examples of what it removed. ``` #### What This Part Should Cover - Which design decisions to present first - Numbers and examples that back each decision - Limitations, and what you would do with more time ### What a Strong Answer Covers - Cleaning rules grounded in observed data, with counts for every rule - A correct single pass with bounded memory and streaming output - Deterministic, reproducible output and clear configuration - Code quality suited to a take-home: structure, error handling and tests - A walk-through that explains trade-offs instead of only listing features ### Follow-up Questions - The corpus grows from about a million documents to a billion. What changes in the design? - How would you catch near-duplicates, such as the same article surrounded by different navigation text, within one pass? - How would you find out whether your filters are removing good documents? - The cleaned data will be used to train a language model. Which additional steps would you consider, and why?