Design ML system for self-driving perception

You are interviewing for a Senior Machine Learning Engineer role on a self-driving car team. They ask you to design a machine learning system for obstacle detection and collision avoidance in urban driving.

The system should run on-vehicle and must:

• Take input from multiple sensors (e.g., cameras, LiDAR, radar).
• Detect relevant objects (cars, pedestrians, cyclists, traffic cones, etc.) and free space.
• Support real-time decision-making to help avoid collisions.

Latency and reliability requirements (you may refine/adjust them during clarifying questions):

• End-to-end perception latency: ≤ 100 ms per frame.
• Very high recall on obstacles within 50 meters in front of the car.
• System must be robust to weather, lighting, and rare edge cases.

Design a complete ML system, covering at least:

1. Problem definition and metrics
   • What exactly the system should output.
   • Offline and online metrics you would use.
2. Data and labeling
   • What data you will collect from the fleet.
   • How you will label obstacles, free space, and other relevant signals.
   • How you will handle class imbalance and rare events.
3. Modeling choices
   • What model architectures you would use for perception (e.g., object detection, segmentation, sensor fusion).
   • How you would fuse multi-modal sensor information.
4. Training pipeline
   • Data preprocessing.
   • Train/validation/test splitting strategy.
   • Handling distribution shift (e.g., new cities, weather conditions).
5. Serving / inference architecture
   • How models are deployed on the car.
   • How you meet latency and reliability constraints.
   • Any model compression or optimization techniques.
6. Monitoring, feedback loop, and continuous improvement
   • How you monitor model quality in production.
   • How you detect and mitigate degradation or new failure modes.
   • How you use new data to retrain and update models safely.

Walk through your design step by step, explaining trade-offs and key decisions.