Design a High-Volume Log Storage System with Live Tailing of Growing Files

Design a storage system for a massive volume of log records. Producers continuously append log records to log files; the system must store them durably and let users read the files back. Some users read a file while it is still being written and expect to see new records as they arrive. The interviewer started from the basic design and kept adding questions about write traffic, partitioning, locating data, write efficiency, live readers and scale. Answer them as the parts below. ### Constraints and Clarifications - Reads fetch a file directly: read it from the beginning, or continue reading where you left off. There are no range queries or searches over record contents. - Write volume, record size, retention and freshness targets are not given; ask for them or state your assumptions. ### Clarifying Questions - Who writes the logs, how many records per second arrive at peak, and how large is a typical record? - How are files named, and how long does a file stay open for writing? - Once a write is acknowledged, may it ever be lost, for example if a machine crashes seconds later? - How quickly must a new record become visible to someone reading the file? - How long are logs kept, and how often are old files read? ### Part 1 — Massive write traffic and partitioning How does the system absorb a massive volume of write traffic, and how should the data be partitioned across storage nodes? ```hint Let the read pattern choose Reads never ask for a range or run a search; they ask for a file. Decide what that means for how you spread data across nodes and what the partition key should be. ``` ```hint One file, many nodes Consider a file that stays open for a long time, or one that receives more writes than a single node can absorb. Decide whether the unit you place on nodes is a whole file or something smaller. ``` #### What This Part Should Cover - A stateless ingestion tier separated from the storage nodes - A partitioning scheme and key justified by the access pattern - Hot files, and adding nodes without large data reshuffles - Replication, and what acknowledging a write guarantees ### Part 2 — Finding the data on read Once a file's data is spread over many data nodes, how does a read know where to look? ```hint Separate names from bytes Compare what a reader knows (a file name) with what it needs (which nodes hold which pieces of the file, and in what order). ``` #### What This Part Should Cover - A metadata layer that maps a logical file to its pieces and their locations - The metadata schema, and how it stays consistent with the data nodes - Keeping metadata off the per-record write path, and making it scale and survive failures ### Part 3 — Making heavy writes efficient Under very heavy write load, how do you improve write throughput? ```hint Many tiny writes Each log record is small. Think about what writing records one at a time costs the network and the disk, and what you would trade to avoid that cost. ``` #### What This Part Should Cover - Batching on producers and servers, with an append-only, sequential layout on disk - Compression and fewer round trips - The durability and latency cost of buffering, and when a write is acknowledged - Backpressure when storage falls behind ### Part 4 — Readers of a file that is still being written A file is still receiving writes. How do users who are reading it see newly appended data promptly, and how does the system know which data is new for each reader? ```hint Where each reader stands Every reader has read the file up to some point. Decide what the reader and the server must agree on so that, even across a dropped connection, nothing is skipped or sent twice. ``` ```hint Ask or be told Compare having the client ask repeatedly with having the server send data as it arrives, in latency and in load. ``` #### Clarifying Questions for this Part - Must readers see a record within about a second, or is a delay of several seconds acceptable? - Should a reader be able to see a record that has not yet been replicated? #### What This Part Should Cover - Polling versus a long-lived connection with server push, and their trade-offs - A per-reader position in the file that survives reconnects - Exposing only durable data to readers, in order ### Part 5 — Many readers, and scaling further If every reader holds a long-lived connection, does that put too much load on the servers? How would you scale the system further? ```hint Count the copies A popular file has thousands of live readers. Count how many times each new record is sent and by which machines, then ask whether those machines should be the ones doing it. ``` #### What This Part Should Cover - The real costs of long-lived connections and of per-reader fan-out - Decoupling delivery to readers from ingestion, with the latency trade-off made explicit - Scaling storage, metadata and delivery independently, including tiering of old data ### What a Strong Answer Covers - A design that separates ingestion, storage, metadata and delivery to readers, each scaled on its own - Partitioning and batching choices justified by the stated access pattern of whole-file reads with no range queries - Explicit durability semantics for buffered and replicated writes - Correct incremental delivery to live readers using positions, across reconnects and failures - Rough sizing used to justify node counts, connection counts and storage tiers ### Follow-up Questions - A data node holding the newest data of a file fails while readers are following that file. What do producers and readers see, and how does the system recover? - One service suddenly writes far more logs than usual. How do you stop it from hurting everyone else? - How would you enforce retention and move old files to cheaper storage without breaking reads? - If search over log contents became a requirement later, how would you add it?