##### Question You are interviewing for a new-grad machine learning / data scientist role at ByteDance. Answer the following related machine-learning and LLM questions. 1. **Multimodal deployment under constraints**: You need to deploy a multimodal model (for example text + image, text + video, or audio + text) under strict **GPU memory (VRAM), compute, latency, and cost constraints**. How would you redesign the model and serving system to reduce memory, latency, and cost while preserving acceptable quality? Discuss model-level changes (quantization, distillation, pruning, input/architecture simplification), serving-level changes (batching, caching, efficient kernels, tiered serving), and modality-specific optimizations, and reason about the tradeoffs of each. 2. **Fast video retrieval with captions and embeddings**: Each video in a large corpus already has a caption and one or more precomputed embedding vectors. How would you build a retrieval system that answers user queries quickly while maintaining high recall? Discuss offline preprocessing, index design, approximate nearest neighbor (ANN) search, lexical vs dense (hybrid) retrieval, reranking, freshness / update trade-offs, and the relevance and serving metrics you would use to evaluate it. 3. **Overfitting**: What is overfitting, how would you detect it, and what are the most effective ways to mitigate it in deep learning systems? 4. **Dropout**: Explain the intuition behind dropout, why it can reduce overfitting, and why the common "inverted dropout" implementation keeps the expected activation scale consistent between training and inference. Also mention when dropout may be less effective or even harmful. 5. **Normalization layers**: Compare Batch Normalization, Layer Normalization, Group Normalization, Instance Normalization, and RMSNorm. What statistics do they use, how do training-time and inference-time behaviors differ, and why are some normalization layers preferred in transformers or small-batch settings? 6. **Reinforcement learning for LLM post-training**: Explain how reinforcement learning is used in LLM post-training, especially in RLHF. Describe the typical pipeline from supervised fine-tuning to preference modeling and policy optimization, the role of KL regularization, common failure modes, and how you would evaluate it. You may also contrast PPO-style RLHF with newer preference-optimization approaches such as DPO.

A six-part ByteDance data scientist onsite ML interview covering deploying and tuning multimodal models under GPU/VRAM, compute, and latency constraints; designing fast video retrieval over precomputed captions and embeddings with ANN and hybrid reranking; and deep-learning fundamentals — overfitting detection and mitigation, dropout and inverted-dropout scaling, normalization layers (BatchNorm/LayerNorm/GroupNorm/InstanceNorm/RMSNorm), and reinforcement learning for LLM post-training (RLHF). It tests practical systems tradeoffs alongside theoretical foundations across ML systems, retrieval, and reinforcement learning.

How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Onsite rounds at Bytedance.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Bytedance during technical interviews.

How to deploy and tune multimodal models? | Bytedance Interview Question

Question

You are interviewing for a new-grad machine learning / data scientist role at ByteDance. Answer the following related machine-learning and LLM questions.

Multimodal deployment under constraints : You need to deploy a multimodal model (for example text + image, text + video, or audio + text) under strict GPU memory (VRAM), compute, latency, and cost constraints . How would you redesign the model and serving system to reduce memory, latency, and cost while preserving acceptable quality? Discuss model-level changes (quantization, distillation, pruning, input/architecture simplification), serving-level changes (batching, caching, efficient kernels, tiered serving), and modality-specific optimizations, and reason about the tradeoffs of each.
Fast video retrieval with captions and embeddings : Each video in a large corpus already has a caption and one or more precomputed embedding vectors. How would you build a retrieval system that answers user queries quickly while maintaining high recall? Discuss offline preprocessing, index design, approximate nearest neighbor (ANN) search, lexical vs dense (hybrid) retrieval, reranking, freshness / update trade-offs, and the relevance and serving metrics you would use to evaluate it.
Overfitting : What is overfitting, how would you detect it, and what are the most effective ways to mitigate it in deep learning systems?
Dropout : Explain the intuition behind dropout, why it can reduce overfitting, and why the common "inverted dropout" implementation keeps the expected activation scale consistent between training and inference. Also mention when dropout may be less effective or even harmful.
Normalization layers : Compare Batch Normalization, Layer Normalization, Group Normalization, Instance Normalization, and RMSNorm. What statistics do they use, how do training-time and inference-time behaviors differ, and why are some normalization layers preferred in transformers or small-batch settings?
Reinforcement learning for LLM post-training : Explain how reinforcement learning is used in LLM post-training, especially in RLHF. Describe the typical pipeline from supervised fine-tuning to preference modeling and policy optimization, the role of KL regularization, common failure modes, and how you would evaluate it. You may also contrast PPO-style RLHF with newer preference-optimization approaches such as DPO.

Question

You are interviewing for a new-grad machine learning / data scientist role at ByteDance. Answer the following related machine-learning and LLM questions.

Multimodal deployment under constraints : You need to deploy a multimodal model (for example text + image, text + video, or audio + text) under strict GPU memory (VRAM), compute, latency, and cost constraints . How would you redesign the model and serving system to reduce memory, latency, and cost while preserving acceptable quality? Discuss model-level changes (quantization, distillation, pruning, input/architecture simplification), serving-level changes (batching, caching, efficient kernels, tiered serving), and modality-specific optimizations, and reason about the tradeoffs of each.
Fast video retrieval with captions and embeddings : Each video in a large corpus already has a caption and one or more precomputed embedding vectors. How would you build a retrieval system that answers user queries quickly while maintaining high recall? Discuss offline preprocessing, index design, approximate nearest neighbor (ANN) search, lexical vs dense (hybrid) retrieval, reranking, freshness / update trade-offs, and the relevance and serving metrics you would use to evaluate it.
Overfitting : What is overfitting, how would you detect it, and what are the most effective ways to mitigate it in deep learning systems?
Dropout : Explain the intuition behind dropout, why it can reduce overfitting, and why the common "inverted dropout" implementation keeps the expected activation scale consistent between training and inference. Also mention when dropout may be less effective or even harmful.
Normalization layers : Compare Batch Normalization, Layer Normalization, Group Normalization, Instance Normalization, and RMSNorm. What statistics do they use, how do training-time and inference-time behaviors differ, and why are some normalization layers preferred in transformers or small-batch settings?
Reinforcement learning for LLM post-training : Explain how reinforcement learning is used in LLM post-training, especially in RLHF. Describe the typical pipeline from supervised fine-tuning to preference modeling and policy optimization, the role of KL regularization, common failure modes, and how you would evaluate it. You may also contrast PPO-style RLHF with newer preference-optimization approaches such as DPO.

How to deploy and tune multimodal models?

Quick Overview

How to deploy and tune multimodal models?

Question

Write your answer

How to deploy and tune multimodal models?

Quick Overview

How to deploy and tune multimodal models?

Question

Write your answer