Design a Multimodal Neural Network
Overview
This question evaluates a candidate's competency in designing production-grade multimodal machine learning systems, including architecture choices for text and image encoders, cross-modal fusion strategies, training objectives for joint retrieval and classification, robustness to missing or noisy modalities, and considerations for scalability and low-latency serving. Commonly asked in ML system design interviews to assess both conceptual understanding and practical application of machine learning engineering principles, it falls under the ML System Design domain and probes abilities in model alignment, evaluation metrics, deployment trade-offs, and domain adaptation strategies.
Design Prompt: Multimodal Text–Image Retrieval and Classification
Context
You are building a production system that uses both text (titles/descriptions/queries) and images to support:
- Cross-modal retrieval (e.g., text-to-image and image-to-text search)
- Item classification (e.g., product category or attributes)
Assume you have paired (image, text) examples for training and must serve at scale with low latency.
Requirements
-
Encoders
- Specify the encoder architecture for each modality (text and image), embedding dimensionality, and any projection layers.
-
Fusion strategy
- Choose and justify early fusion, late fusion, or cross-attention (or a hybrid), and describe how features are combined.
-
Training objectives
- Include contrastive alignment for retrieval and a joint classification objective. Detail the loss functions.
-
Missing or noisy modalities
- Describe how to train for and handle cases where a modality is missing or low quality at inference time.
-
Data alignment and augmentation
- Explain how to align training data across modalities and augment it to improve robustness.
-
Evaluation metrics
- Define metrics for retrieval and classification, plus robustness and efficiency metrics.
-
Scalability and serving latency
- Discuss training/inference scale-out, approximate nearest-neighbor search, batching, quantization, and latency budgets.
-
Fine-tuning on new domains
- Outline approaches for rapid domain adaptation with limited labels.
Solution
Login required