Design video captioning under compute limits

Q: Design video captioning under compute limits

This is a ML System Design interview question from TikTok for Machine Learning Engineer roles. View the full question and solution on PracHub.

Q: How do I approach ML System Design interview questions?

ML System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master ml system design interviews.

Question

Scenario

You are deploying a multimodal large model that generates captions for videos.

Part A — Deployment under compute / VRAM constraints

The model takes video (frames/audio optional) and outputs a text caption.
You must meet latency/throughput goals while staying within tight compute and GPU memory (VRAM) limits.

Prompt: Describe how you would design the end-to-end system (modeling + serving) to reliably deploy this capability under constrained compute/VRAM.

Part B — Fast retrieval for brand ads + watermarking

Assume you already have:

A caption for each video (possibly multiple captions per video)
An embedding vector per video (or per caption)

A brand advertiser wants to quickly find videos relevant to a query (text and/or example creative) and then apply a watermark to matched videos.

Prompt: How would you design and optimize the retrieval + processing pipeline to make this search and watermarking fast at scale? Include indexing, filtering/ranking, caching, and system trade-offs.

Design video captioning under compute limits

Scenario

Part A — Deployment under compute / VRAM constraints

Part B — Fast retrieval for brand ads + watermarking

Solution

Comments (0)