Design a Durable Concurrent Event Writer
Company: Databricks
Role: Software Engineer
Category: System Design
Difficulty: hard
Interview Round: Onsite
# Design a Durable Concurrent Event Writer
Design a thread-safe `DataWriter` used by one application process on one server:
```text
DataWriter(file_path)
push(data: bytes) -> None
```
Thousands of application threads share one instance and continuously append opaque records of at most 1 KB to a local file. The records must not interleave or become silently corrupted. Discuss ordering, durability, backpressure, failure reporting, shutdown, and crash recovery. State the exact guarantee provided when `push` returns.
### Constraints & Assumptions
- There is one writer object and one target file.
- Record contents are opaque and may contain arbitrary bytes.
- The process may crash at any instruction and the disk may report write or flush errors.
- You may batch physical writes, but must explain how batching affects acknowledgment semantics.
### Clarifying Questions to Ask
- Must `push` wait for an `fsync`, or is acceptance into process memory sufficient?
- Is ordering defined by call start, lock acquisition, or an assigned sequence number?
- May producers block when the writer is overloaded?
- How should a caller learn about an asynchronous disk failure?
- What should graceful shutdown guarantee?
### What a Strong Answer Covers
```premium-lock What a Strong Answer Covers
```
### Follow-up Questions
- How would you support several independent files?
- How would the design change if every acknowledged record had to survive power loss?
- How would you rotate and compact files safely?
- What changes if several processes append to the same file?
Quick Answer: Design a thread-safe local event writer shared by thousands of producers appending opaque records to one file. Define ordering and return guarantees while addressing interleaving, bounded backpressure, batching, disk errors, torn writes, graceful shutdown, crash recovery, and operational metrics.