System Design: High-Throughput Content Processing Service

Context

Design a content processing and moderation service for a large-scale media app. Content includes images and short videos uploaded by users. The service must ingest content reliably, classify it using ML, apply policy rules, and route outcomes for auto-action or human review.

Assume 10M items/day on average with bursty traffic (e.g., 10× spikes around events). Near-real-time means prioritized items should be fully processed within minutes, and non-priority can tolerate longer latency.

Requirements

1. Ingestion
   • Ingest 10M items/day with bursty traffic.
   • Handle authentication, idempotency, and client-side retries.
2. Processing Pipeline
   • Near-real-time ML classification and rule evaluation.
   • Priority queues (e.g., P0, P1) and backpressure so bursts do not overload the system.
3. Storage
   • Choose storage for media (binary) and metadata (records, decisions, audit logs).
4. APIs
   • Expose APIs for upload, review (moderator tooling), and appeals.
5. Experimentation
   • Add A/B testing hooks to try multiple models/rules and support offline evaluation.
6. Non-Functional
   • Scalability, fault tolerance, observability, and cost efficiency.

Deliverables

• Provide a high-level architecture diagram.
• Justify trade-offs and key technology choices.