Search Words and Phrases in a Large Line-Oriented File

# Search Words and Phrases in a Large Line-Oriented File A text file contains one sentence per line and may be much larger than memory. Design an API that first supports finding every line containing a word and then supports finding every line containing an exact multiword phrase. Discuss both a one-off search and a system that serves repeated queries against a mostly stable file. Define tokenization, phrase boundaries, case handling, result format, and how updates become searchable. ### Constraints & Assumptions - Sequential reads are allowed; loading the complete file into memory is not. - Results identify at least the line number and may include byte offsets. - Exact behavior for punctuation and Unicode must be stated. - The repeated-query design may use persistent auxiliary storage. ### Clarifying Questions to Ask - Is a word matched as a token or as an arbitrary substring? - Must phrase terms be adjacent and in the same line? - Are queries rare enough for a full scan, or is low query latency required? - Can the source file be appended to or rewritten while indexing? ### What a Strong Answer Covers - A correct streaming scan for one-off queries - Token and phrase semantics that remain consistent between indexing and lookup - An inverted index with positions for repeated phrase queries - Memory, CPU, storage, and update complexity - Safe file-version binding, malformed input handling, and tests ### Follow-up Questions - How would you return matches in relevance order rather than file order? - How would you shard an index that no longer fits on one machine? - What changes for compressed input? - How can a new index version become visible atomically?