System Design: Image Object Detection Service

Scenario

Design an image detection service that accepts user-uploaded images and returns detected objects with bounding boxes and confidence scores. The service must support both real-time online inference and high-throughput batch processing.

Clarify and Quantify Requirements

State assumptions if any numbers are missing, and justify them.

1. Traffic and performance

• Online inference: target end-to-end latency (P50/P95/P99), expected and peak QPS, regional distribution, and acceptable tail behavior.
• Batch processing: target throughput (images/sec), acceptable end-to-end SLA (e.g., minutes/hours), and concurrency.
• Payloads: typical and max image size (KB/MB), formats (JPEG/PNG/WebP), and max resolution.

2. Quality

• Accuracy metrics: required mAP@0.5 and mAP@[0.5:0.95], precision/recall, calibration targets.
• Classes: expected number of object classes and class imbalance considerations.

3. Constraints

• Cost budget, multi-tenancy/isolation needs, regions, and compliance requirements.

Deliverables

Propose and justify an end-to-end architecture that includes:

1. Ingestion

• API gateway, auth (e.g., OAuth2/JWT), WAF, rate limiting/quotas, request validation.
• Upload flow: direct-to-object-store with pre-signed URLs vs. through API.

2. Storage and data model

• Object storage for raw/derived images; metadata database schema for requests, results, and model versions.

3. Preprocessing and postprocessing

• Image validation, resizing/normalization, EXIF handling, virus scanning, and output formatting.

4. Model serving

• GPU-backed serving, autoscaling policy, batching strategy, model formats (e.g., ONNX/TensorRT), and multi-model/version hosting.

5. Caching

• Result caching strategy (keys, TTL), CDN considerations, and cache invalidation on model updates.

6. Asynchronous workflows

• Queues/streams, idempotency, retries/DLQs for batch and overflow traffic.

7. Model lifecycle

• Versioning, A/B testing or shadow traffic, rollout/rollback.

8. Monitoring and reliability

• Metrics (latency percentiles, throughput, GPU utilization/memory, queue depth), drift detection, alerting SLOs, failure modes and fallbacks.

9. Offline training pipeline

• Data labeling, augmentation, experiment tracking, evaluation, and packaging for serving.

10. Deployment and rollout

• Blue/green or canary strategy; CI/CD and validation gates.

11. Cost/performance trade-offs

• GPU types, batching, quantization, spot vs. on-demand, and multi-tenancy isolation.

12. Privacy, security, compliance

• Data retention/deletion, encryption, access control, audit logging, regionalization, and acceptable use/content controls.

Provide diagrams if useful (ASCII is fine), capacity planning math, and a brief risk/mitigation section.