Design a real-time 3D playback system for recorded vehicle logs

Design a system that plays back recorded vehicle log data as an interactive 3D scene. Each recorded drive is a set of logs of time-stamped data stored in object storage, such as an S3 bucket. An engineer opens a drive in a viewer and watches the vehicle and its recorded data move through a 3D scene in real time, with play, pause and a timeline to seek on. Playback must start quickly and stay smooth, even though one recorded drive can be far too large to download before the first frame is shown. ```hint Think like a video player A video site does not make you download a whole movie before the first frame plays. Consider what the equivalent of a video segment and a playlist would be for time-stamped log data. ``` ```hint Seeking into the middle A user drags the timeline to the middle of a long drive. Decide exactly what the viewer must fetch to draw that moment correctly, and what it should fetch next. ``` ```hint Many streams, one clock Different kinds of recorded data are sampled at different rates. Decide how the viewer chooses what to draw for a given playback time. ``` ### Constraints and Clarifications - The logs already exist in object storage when playback is requested; the system reads them and does not control how they were recorded. - Assume every recorded message carries a timestamp, so all data in a drive can be placed on one timeline. ### Clarifying Questions - What does a log contain: only the vehicle's pose over time, or also heavy sensor data such as point clouds and camera images, and at what rates? - How large is one recorded drive, and in what file format is it stored? - Does "real time" mean smooth playback of recorded logs at recorded speed, or must the viewer also follow a drive that is still being uploaded? - How many engineers watch playback at once, and do many of them open the same drives? - Does playback need speeds other than 1x, frame-by-frame stepping, or playing backward? - Must the viewer run in a web browser? ### What a Strong Answer Covers - Fetching a drive in time-ordered chunks on demand instead of downloading the whole log before playback - A preprocessing step that turns raw logs into seekable, time-indexed chunks with a manifest, and the chunk-size trade-off - A client-side buffer and cache that prefetches upcoming timestamps and keeps nearby ones for scrubbing, including what happens on a seek - One playback clock that aligns data streams of different rates, with interpolation for smooth motion - Handling heavy data (decimation or levels of detail), caching immutable chunks at the edge, and failure handling for missing or slow chunks - Measuring time to first frame, seek latency and playback stalls ### Follow-up Questions - How would you support playback at 4x speed, or stepping backward one frame at a time? - How would the design change if engineers must watch a drive while the vehicle is still uploading it? - Two engineers want to review the same moment together and always see the same frame. How would you synchronize their viewers? - How would you let users search across many drives for moments of interest and jump straight to them?