Design distributed log storage service

Quick Overview

This question evaluates a candidate's expertise in designing distributed storage systems, covering partitioned append-only logs, high-throughput writes and reads, replication, retention, indexing, compaction, leader/follower roles, and recovery mechanisms.

Design distributed log storage service

Company: Google

Role: Software Engineer

Category: System Design

Difficulty: hard

Interview Round: Technical Screen

How would you design storage for distributed data logs supporting append-only writes, retention policies, partitioning, replication, and high-throughput sequential reads? Detail segment management, compaction, indexing, leader and follower roles, and recovery.

Overview: This question evaluates a candidate's expertise in designing distributed storage systems, covering partitioned append-only logs, high-throughput writes and reads, replication, retention, indexing, compaction, leader/follower roles, and recovery mechanisms.

|Home/Google/System Design
Google logo
Google
Sep 6, 2025
hardSoftware EngineerTechnical ScreenSystem Design
19
0

Design a Distributed Append-Only Log Storage System

You are asked to design the storage layer of a distributed, partitioned, replicated append-only log service that supports:

  • Append-only writes with high throughput
  • Retention policies (by time and/or size)
  • Partitioning for horizontal scalability
  • Replication for fault tolerance
  • High-throughput sequential reads

Detail the following components and behaviors:

  1. Segment management: active vs. sealed segments, rollover conditions, deletion, and preallocation.
  2. Compaction: when and how to compact, tombstones/deletes, write amplification tradeoffs.
  3. Indexing: offset-to-file-position index, time index, sparsity, and rebuild.
  4. Leader/follower roles: write and read paths, quorum/ack semantics, high watermark, lag handling.
  5. Recovery: crash recovery, index rebuild, truncation, leader election safety, and follower catch-up.

Assume a large-scale multi-node deployment with commodity disks and network, and that clients produce and consume records identified by monotonically increasing offsets within each partition.

Browse More Questions

More System Design•More Google•More Software Engineer•Google Software Engineer•Google System Design•Software Engineer System Design