# Design a Compact Columnar File Format Design a simplified Parquet-like file format and specify `write` and `read` operations. The format should store typed columns, support encoding and compression choices, detect corruption, and allow a reader to retrieve selected columns without decoding the entire file. Explain file layout, metadata, compatibility, and failure handling. ### Constraints & Assumptions - Support integers, floating-point values, strings, booleans, and nulls. - A reader may request a subset of columns and a row range. - Files can be larger than memory, so writing and reading must stream bounded chunks. - Unknown future metadata must be skippable by older readers when compatibility permits. ### Clarifying Questions to Ask - Is random row lookup required, or are scans and ranges sufficient? - Which schema changes must old readers accept? - Should a corrupted chunk make the whole file unreadable or only that chunk unavailable? ```hint Make chunks self-describing A reader needs offsets, lengths, encoding, compression, null information, and a checksum before decoding bytes. ``` ```hint Plan finalization If offsets are learned while streaming, put authoritative metadata in a footer located by a fixed trailer. ``` ### What a Strong Answer Covers - A byte-level layout with header, schema, column chunks, indexes, footer, and checksums. - Encoding choices such as dictionary, run-length, delta, and plain encoding tied to data properties. - Streaming write finalization and projected read behavior. - Versioning, bounds checks, corruption handling, and test strategy. ### Follow-up Questions - How would min and max statistics enable row-group pruning? - How would you avoid unbounded allocation when reading a malicious file?