Design line-preserving file chunker pipeline
Company: Google
Role: Software Engineer
Category: System Design
Difficulty: hard
Interview Round: Technical Screen
Design a data pipeline that reads many text files of varying sizes and emits output files of exactly 100 MB each, preserving line boundaries and ensuring every input line appears exactly once in the output. Specify how to handle partial lines at boundaries, buffering, compression, parallelism, and fault tolerance with exactly-once output without duplicates or omissions.
Overview: This question evaluates expertise in designing distributed data pipelines with precise file chunking, line-boundary preservation, exactly-once semantics, buffering and partial-line handling, compression trade-offs, parallelization, and fault tolerance.