Design an IoT Logging Platform with Late-Arriving Metrics

Design a logging platform for a large fleet of devices. Devices must continue collecting logs while online or offline and upload batches when connected. The platform must also be able to request that a device collect specified types of logs. The platform reliably receives and stores logs and reports success and failure rates by log type. Some devices may reconnect together after an offline period, so late arrivals must update the reports for the periods to which those logs belong. ### Constraints and Clarifying Questions - Clarify the fleet size, log volume, maximum supported offline interval, local storage budget, and acceptable ingestion and reporting delay. No numerical targets are supplied. - Define what a success or failure means for each log type and what belongs in each rate's denominator. - Clarify whether report periods follow event time or upload time, and how much confidence the platform can place in device clocks. - Clarify whether a collection request selects a temporary capture or a persistent configuration, and what should happen if it expires while the device is offline. ### Part 1 — Collect Offline and Upload Reliably Describe the device-side log representation, buffering, upload protocol, and acknowledgment behavior. Explain how the design behaves after a device restart, a lost acknowledgment, or many devices reconnecting at once. #### What This Part Should Cover - Stable event identity and sufficient metadata to classify and place each event in time. - Durable local buffering, retransmission, duplicate handling, and a defined storage-limit policy. - An ingestion boundary that acknowledges only the durability it actually provides and applies backpressure during reconnection bursts. ### Part 2 — Request Particular Log Types Design the control path from a platform request to the device applying the requested collection behavior. Account for offline devices and separate request acceptance from evidence that a device has acted on the request. #### What This Part Should Cover - Durable per-device command or configuration state and how a reconnecting device obtains it. - Request identity, versioning, expiration, and repeated delivery. - Observable states for requested, received, applied, or failed collection changes. ### Part 3 — Maintain Reports as Late Logs Arrive Describe how to store the logs and derive success and failure rates by type. Explain how a batch uploaded after a long offline interval changes an existing historical report without double-counting retried events. #### What This Part Should Cover - Event-time attribution, durable raw data, deduplication, and aggregate update behavior. - Correct numerators and denominators, including empty or incomplete periods. - Report freshness, revision visibility, and repair or recomputation when arrivals exceed the usual lateness window. ```hint Follow one delayed event Track a log collected offline, uploaded twice because its first acknowledgment was lost, and received after a report was already displayed. Identify each place where it could be lost, counted twice, or assigned to the wrong period. ``` ### What a Strong Answer Covers - A complete data path and control path with distinct delivery guarantees. - Failure recovery across device storage, durable ingestion, processing, and report updates. - Explicit limits on offline retention and report completeness instead of an unsupported promise that every displayed historical value is final. ### Follow-up Questions - How would the design handle a device whose clock was wrong throughout its offline interval? - What should a report show when no events of a type have arrived, or when many devices are still offline? - How would you repair one time bucket after discovering a processing defect without disrupting newer ingestion?