System Design: End-to-End Image Upload Pipeline
Context
Design an end-to-end system that handles user picture uploads from web or mobile clients through the backend and storage/CDN layers. Assume an object store (e.g., S3/GCS), a CDN, an API service, a background processing tier, and a relational database. Images are typically user-generated (e.g., profile or listing pictures), with typical sizes up to 20 MB, and must be delivered globally.
Requirements
Describe, in detail, how the system handles:
-
Request flow from browser/app through backend and storage/CDN.
-
Authentication and authorization.
-
Upload strategies: direct-to-object-store vs. server-proxied.
-
Chunked uploads and resume support.
-
Client- and server-side validation (file type/size, checksums, virus scanning).
-
Metadata extraction (e.g., EXIF) and sanitization.
-
Asynchronous processing pipelines/queues.
-
Image processing (resize/thumbnailing, formats, orientation).
-
Storage strategy and versioning, including deduplication and lifecycle.
-
Database writes and idempotency.
-
CDN caching and invalidation strategies.
-
Error handling and retries across components.
-
Observability and alerting.
-
Performance and cost considerations.
-
Horizontal scaling and multi-region considerations.
Clearly explain trade-offs and provide guardrails for safe operation. Use small numeric examples where helpful.