Single-Pass Python Cleaning Pipeline for Million-Document Web Crawl Parquet Files
Company: xAI
Role: Data Engineer
Category: Data Manipulation (SQL/Python)
Difficulty: medium
Interview Round: Onsite
Overview: A data engineering take-home: write one Python script that reads Parquet files holding about a million crawled web documents in a single pass and writes cleaned output, then walk a hiring manager through it. It tests cleaning rules, deduplication, streaming within a memory budget, and explaining trade-offs.