Design a Kafka-Based File Watcher for Millions of Files

Design a file-watcher system that handles millions of files and uses Kafka to deliver change events to downstream consumers. Explain initial discovery, ongoing changes, ordering, recovery, and how the system detects changes missed by native filesystem notifications. ### Constraints & Assumptions - Clarify whether files are on local disks, many hosts, or a shared filesystem, and what event guarantees those sources provide. - Define the desired result: every low-level event, or eventual convergence to the current file state. Those are different guarantees. - The source specifies scale in file count but no event rate, file size, or latency target. ### Clarifying Questions to Ask - Which changes matter: creation, content updates, deletion, rename, or metadata changes? - Is there a durable source journal, or can watch events overflow or disappear during downtime? - Is ordering required per file, per directory, per host, or globally? - Do consumers need file contents or only metadata and a versioned reference? ```hint Reconcile after a gap A durable Kafka log cannot recover a filesystem event that was lost before it reached the producer. Define how source state is checked when notification continuity is uncertain. ``` ### What a Strong Answer Covers - Sharded watcher agents, initial scan/watch coordination, and resource limits. - Stable file identity, event versions, rename handling, and partition keys. - Durable ingestion, consumer replay, idempotency, and per-key ordering. - Periodic reconciliation or journal recovery for missed notifications. - Backpressure, hot partitions, and operational lag/coverage metrics. ### Follow-up Questions - How would an event overflow during a burst of changes be detected and repaired? - What happens if a file changes again before a consumer reads its contents?