Design an Autonomous-Vehicle Perception Pipeline

# Design an Autonomous-Vehicle Camera Perception Pipeline Design the camera and perception pipeline for an autonomous vehicle. Trace data from physical cameras through capture, synchronization, calibration, preprocessing, model inference, multi-camera fusion, tracking, and the interface to downstream planning. Explain how the design meets real-time latency, safety, reliability, and observability requirements on constrained onboard hardware. ### Constraints & Assumptions - Several cameras have different fields of view and produce high-bandwidth streams. - Decisions have a hard end-to-end latency budget; stale output can be unsafe. - Compute, memory bandwidth, power, and cooling are limited. - Individual frames, cameras, accelerators, or software components may fail. - Models and calibration data change over the vehicle's lifetime. ### Clarifying Questions to Ask - Which cameras, frame rates, resolutions, and environmental conditions are supported? - What objects, lanes, free space, and uncertainty must the pipeline output? - What latency deadline and degradation policy apply? - Is camera-only operation required, or can the pipeline fuse radar or lidar? ### What a Strong Answer Covers - Timestamping, synchronization, calibration, image validation, and bounded buffering - A staged real-time dataflow with accelerator-aware scheduling and backpressure - Detection or segmentation, cross-camera fusion, temporal tracking, and uncertainty - Health monitoring, redundancy, degraded modes, safe failure behavior, and replay - Versioned models and calibration, shadow rollout, metrics, and fleet feedback ### Follow-up Questions 1. What should happen when one front-facing camera stops producing valid frames? 2. How would you detect that a calibration change has silently degraded geometry? 3. Which stages can be skipped or reduced when the accelerator is overloaded? ```hint Make freshness an explicit invariant Bound every queue and attach capture time and version metadata so downstream consumers can reject stale or incompatible results. ```