Design an Autonomous-Vehicle Camera Pipeline and Its Perception System

Design the camera pipeline of an autonomous vehicle and the perception system that consumes it. The pipeline starts at the vehicle's cameras and ends with perception outputs, such as detected objects and scene information, delivered to downstream consumers like planning. The role is a senior integration position on an autonomous-driving team whose stack is written in C++, so the interviewer expects an architecture discussion grounded in how the pieces run on the vehicle, not model-training details or code. ### Clarifying Questions - How many cameras are there, and with what resolutions, frame rates and fields of view? - What end-to-end latency is required, measured from exposure at the sensor to the perception output, and at what rate must outputs be produced? - What compute is on the vehicle: one system-on-chip with a GPU and hardware image processing blocks, or several computers? - Which perception outputs are required (2D or 3D objects, lanes, free space, traffic lights), and are radar or lidar fused in this system or elsewhere? - What must the system do, and report, when a camera fails or its view is blocked? - Must raw or processed data be recorded on the vehicle for debugging and replay? ### Part 1 — The camera pipeline Design the path from the cameras to frames that are ready for perception: capture, timestamping and synchronization across cameras, image signal processing, buffer and memory management, and delivery to consumers. Estimate the data rates involved and explain what happens when a downstream stage falls behind. ```hint Follow one frame Trace a single frame through the pipeline: what timestamp it carries, where its memory lives at each stage, and who is allowed to hold it for how long. ``` #### What This Part Should Cover - Capture, a common time base, and synchronization of exposures across cameras - Image signal processing and the choice of formats handed to perception - Zero-copy buffer ownership, bounded queues, and a policy for frames that cannot be processed in time - A data-rate estimate that drives the design ### Part 2 — The perception system Design the perception system that consumes the synchronized camera frames: how models run across all cameras on shared compute, how per-camera results become objects in the vehicle's frame of reference, how they are tracked over time, and what exactly is delivered to planning. Include how the system behaves when a camera degrades or stops. ```hint Budget, then degrade Split the latency budget across the stages first, then decide what a consumer should receive in a cycle where one camera has no fresh frame. ``` #### What This Part Should Cover - Model execution across cameras (batching, precision, scheduling) within a latency budget - Transformation to a common vehicle frame, fusion across overlapping cameras, and tracking - The output contract to planning, including timestamps, uncertainty and sensor health - Fault detection and degraded modes ### What a Strong Answer Covers - Requirements and bandwidth and latency estimates stated before the design - A consistent time model: hardware timestamps, a synchronized clock, and ego-motion compensation - Real-time C++ engineering: preallocated buffers, bounded queues, no allocation or blocking on the hot path - Clear interfaces between capture, processing, perception and planning - Failure detection, degraded operation, and the observability and record-and-replay tooling needed to integrate and debug the system ### Follow-up Questions - One camera's timestamps start drifting relative to the others. How do you detect it, and what does perception do until it is fixed? - How would you design record-and-replay so that a problem seen on the road can be reproduced deterministically on a workstation? - A new detection network takes twice as long to run. Where do you find the budget, and what do you give up? - How do you validate an updated camera calibration before it is deployed to vehicles?