Design device logging system with offline upload
Quick Overview
This System Design question evaluates a candidate's ability to architect a distributed logging and ingestion system that supports offline buffering, reliable uploads, and near-real-time visibility; it belongs to the System Design/distributed systems domain and requires architectural-level reasoning about end-to-end data flows, APIs, storage, and UI integration. It is commonly asked because interviewers use it to assess scalability and reliability thinking under intermittent connectivity, burst traffic handling, ordering and deduplication concerns, latency guarantees, and trade-offs between consistency, durability, and operational cost.
Design a logging system that collects logs from many client devices, uploads them to the backend, and makes them viewable in a web UI within 3–5 minutes of being generated (when connectivity allows).
Requirements:
- Devices generate logs continuously (structured or unstructured text).
- Devices may be offline for extended periods; when they reconnect, they must upload accumulated logs reliably.
- Provide a UI (and supporting APIs) to browse/search/filter logs by device, time range, severity, etc.
- Target freshness: logs should appear in the UI within 3–5 minutes after being generated if the device is online; if offline, they should appear within ~3–5 minutes after reconnection and successful upload.
Follow-ups:
- How does your design handle devices that have no internet connectivity for a while (offline buffering, reliability, deduplication, ordering)?
- If many devices come online and start uploading at the same time (e.g., 6 PM daily), how does your design handle the traffic surge (throttling, autoscaling, load shedding, prioritization)?