Design multimodal deployment under compute limits

You need to answer a set of questions related to multimodal model deployment and post-training optimization in an interview. Provide systematic explanations based on engineering feasibility and ML principles (you may use bullet points or mini-frameworks).

1) How to Deploy Multimodal Models Under Compute and Memory Constraints

Assume you need to deploy a multimodal model (e.g., image-text/video-text retrieval or understanding model) in a resource-constrained environment (possibly a single mid-range GPU or edge device), with the goal of providing stable service at acceptable latency and cost.

Please explain:

• How would you approach end-to-end inference optimization and system design (covering both model-side and system-side)?
• What are common strategies for dealing with GPU memory bottlenecks vs. compute bottlenecks?
• How do you make trade-offs between quality, latency, and cost , and what offline/online metrics and monitoring do you need for regression validation?

2) How to Speed Up Video Retrieval with Existing Captions and Embeddings

You have already generated the following offline for videos:

• caption : text descriptions of videos or video segments
• embedding : vectors for semantic retrieval (may include text/visual/multimodal vectors)

At query time, given a user query (primarily text), you need to return Top-K videos (or segments) with low latency and high throughput.

Please explain:

• How to design a two-stage/multi-stage retrieval architecture for acceleration (e.g., candidate recall + fine ranking/re-ranking).
• How to optimize on the vector retrieval side: ANN indexing, sharding, compression, caching , etc.
• How to do hybrid retrieval combining captions and embeddings, and potential failure modes (e.g., semantic drift, popularity bias, insufficient long-tail recall).

3) What Is Overfitting? How to Mitigate It?

• Define overfitting (from the perspectives of training/validation error, generalization, and model capacity).
• Provide at least 5 categories of common mitigation techniques, and explain their applicable scenarios and side effects.

4) Dropout Principles and Inference-Time Handling

• Explain what Dropout does during training.
• Why is scaling needed (to maintain distribution/expectation consistency)?
• How is Dropout handled during inference (and how does it differ from training)?

5) Compare Different Normalization Methods and Explain Inference-Time Handling

Compare and explain the core differences and applicable scenarios of at least the following normalization methods:

• BatchNorm (BN)
• LayerNorm (LN)
• GroupNorm (GN) / RMSNorm (choose one or more)

Also answer:

• What statistics/formulas does each use during inference?
• What issues may arise with small batches, distribution shift, or mixed precision, and how to mitigate them?

6) Reinforcement Learning in LLM Post-Training (RLHF)

Outline the typical RLHF pipeline and key components:

• Preference data and the Reward Model
• Policy optimization (e.g., PPO-based methods) and KL constraints

Also discuss:

• Benefits and common risks of RLHF (reward hacking, alignment tax, degeneration, etc.).
• Possible alternatives (e.g., DPO/IPO, RLAIF, best-of-N/rejection sampling, etc.) and their trade-offs.