Design a content processing service
Overview
This question evaluates understanding of designing scalable, reliable content-processing systems that handle high-throughput ingestion, ML-based classification and rule evaluation, priority/backpressure mechanisms, storage and API design, experimentation hooks, and operational concerns like observability and cost.
System Design: High-Throughput Content Processing Service
Context
Design a content processing and moderation service for a large-scale media app. Content includes images and short videos uploaded by users. The service must ingest content reliably, classify it using ML, apply policy rules, and route outcomes for auto-action or human review.
Assume 10M items/day on average with bursty traffic (e.g., 10× spikes around events). Near-real-time means prioritized items should be fully processed within minutes, and non-priority can tolerate longer latency.
Requirements
-
Ingestion
- Ingest 10M items/day with bursty traffic.
- Handle authentication, idempotency, and client-side retries.
-
Processing Pipeline
- Near-real-time ML classification and rule evaluation.
- Priority queues (e.g., P0, P1) and backpressure so bursts do not overload the system.
-
Storage
- Choose storage for media (binary) and metadata (records, decisions, audit logs).
-
APIs
- Expose APIs for upload, review (moderator tooling), and appeals.
-
Experimentation
- Add A/B testing hooks to try multiple models/rules and support offline evaluation.
-
Non-Functional
- Scalability, fault tolerance, observability, and cost efficiency.
Deliverables
- Provide a high-level architecture diagram.
- Justify trade-offs and key technology choices.
Solution
Login required