Design a Multimodal Training Data Pipeline

Design a backend system for collecting, filtering, and storing training data sent by many clients. Clients upload records that may include large media files such as images or videos, action sequences, labels, and metadata. The backend must run a series of validation and filtering steps, including: - Detect whether an image or video frame is upside down or otherwise invalid. - Detect duplicate or near-duplicate actions or sequences. - Detect spam, junk, or low-quality submissions. - Store accepted data for downstream machine learning training. Discuss the end-to-end system design, including APIs, ingestion flow, storage, asynchronous processing, scalability, reliability, handling large media files, observability, and how you would detect and mitigate data bias in the collected training data.