Design Prompt: Multimodal Text–Image Retrieval and Classification

Context

You are building a production system that uses both text (titles/descriptions/queries) and images to support:

• Cross-modal retrieval (e.g., text-to-image and image-to-text search)
• Item classification (e.g., product category or attributes)

Assume you have paired (image, text) examples for training and must serve at scale with low latency.

Requirements

1. Encoders
   • Specify the encoder architecture for each modality (text and image), embedding dimensionality, and any projection layers.
2. Fusion strategy
   • Choose and justify early fusion, late fusion, or cross-attention (or a hybrid), and describe how features are combined.
3. Training objectives
   • Include contrastive alignment for retrieval and a joint classification objective. Detail the loss functions.
4. Missing or noisy modalities
   • Describe how to train for and handle cases where a modality is missing or low quality at inference time.
5. Data alignment and augmentation
   • Explain how to align training data across modalities and augment it to improve robustness.
6. Evaluation metrics
   • Define metrics for retrieval and classification, plus robustness and efficiency metrics.
7. Scalability and serving latency
   • Discuss training/inference scale-out, approximate nearest-neighbor search, batching, quantization, and latency budgets.
8. Fine-tuning on new domains
   • Outline approaches for rapid domain adaptation with limited labels.