System Design: Image Object Detection Service

Design an image detection system that processes user-uploaded images and returns detected objects with bounding boxes and confidence scores.

Clarify and Propose Requirements

• Online inference SLOs: propose latency targets (p50/p90/p95), availability, and error budgets.
• Batch throughput: target items/hour and max end-to-end time for large jobs.
• Expected traffic: peak and average QPS, daily active users, regional traffic patterns.
• Payloads: supported formats (JPEG/PNG/WebP/HEIC/GIF), max image size (MB, dimensions), typical sizes, and content-type constraints.
• Accuracy metrics: mAP@0.5 and mAP@[0.5:0.95], per-class precision/recall, calibration (ECE), and confidence thresholds.

Architecture Scope

Propose an end-to-end architecture covering:

1. Ingestion: API/gateway, authentication/authorization, rate limiting, request validation.
2. Storage: object store for images, metadata database for detections and jobs.
3. Preprocessing: format normalization, resizing, EXIF handling, validation, augmentation for offline.
4. Model serving: GPU-backed inference, dynamic batching, autoscaling.
5. Caching: result caching keyed by content-hash and model version.
6. Asynchronous workflows: job API, queues, workers for batch.
7. Model lifecycle: versioning, A/B testing, shadow traffic.
8. Monitoring/observability: latency, throughput, GPU utilization, data drift, accuracy.
9. Failure handling and retries.
10. Offline training pipeline: data labeling, augmentation, experiments, evaluation.
11. Deployment strategy: blue/green or canary.
12. Cost/performance trade-offs.
13. Privacy, security, and compliance.