System Design: Distributed Word-Frequency Computation (No MapReduce)

Context

Design a distributed system that computes word frequencies over terabytes of text data without using MapReduce. Your design should support both batch (historical backfills) and streaming (near real-time) modes and scale horizontally.

Requirements

Describe and justify your approach for the following:

1. Ingestion
   • Batch input from large files in object storage.
   • Streaming input from event/log pipelines.
2. Partitioning & Routing
   • Hash-by-token with consistent hashing and virtual nodes.
   • Handling hot keys and data skew (e.g., stop-word mitigation, heavy-hitter routing).
3. Intermediate Storage
   • Durable buffering between stages; topic/queue design; compaction vs retention.
4. Aggregation Topology
   • Combiner layer, key ownership, state storage (e.g., local state stores), and how updates flow.
5. Processing Semantics
   • Exactly-once or at-least-once guarantees; idempotency strategy.
6. Fault Tolerance & Replay
   • Checkpointing state and offsets, recovery, reprocessing.
7. Backpressure & Flow Control
   • How slow consumers protect the system; throttling strategies.
8. Horizontal Scaling & Rebalancing
   • Adding/removing nodes; consistent hashing and state migration.
9. Monitoring & Operations
   • Metrics, alerts, tracing, skew detection, capacity signals.
10. SLA/SLO Considerations
    • Latency/throughput targets for streaming; completion targets for batch; error budgets.
11. Batch vs Streaming Trade-offs
    • Contrast the two approaches and justify your design choices.