Design a scalable video search system

Read the full interview experience this question came from →

Quick Overview

This interview question evaluates ML product requirements, data/labeling, modeling, serving architecture, evaluation, monitoring, and trade-offs in a realistic interview setting. A strong answer for Design a scalable video search system states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design a scalable video search system

Company: Runway

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Technical Screen

Design an end-to-end video search system that supports text-to-video and video-to-video retrieval. Specify the model architecture (e.g., dual encoders for video frames/clips and text, temporal aggregation strategies, pretraining/fine-tuning objectives, negative sampling, handling long videos via clip sampling), the shared embedding space, and relevance scoring. For infrastructure, describe the offline/online feature extraction pipeline, ingestion, metadata enrichment, indexing in a vector store (e.g., HNSW/IVF, sharding and replication), storage layout for thumbnails/clips, query routing, latency/cost targets, caching, and scalability for billions of clips. Include monitoring, abuse/content safety, and evaluation via offline metrics and online A/B tests; discuss relevance feedback and incremental model updates.

Overview: This interview question evaluates ML product requirements, data/labeling, modeling, serving architecture, evaluation, monitoring, and trade-offs in a realistic interview setting. A strong answer for Design a scalable video search system states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Read the full Runway Machine Learning Engineer interview experience this question came from

|Home/Runway/ML System Design
Runway logo
Runway
Aug 8, 2025
hardMachine Learning EngineerTechnical ScreenML System Design
20
0

Design a scalable video search system

Design a Text-to-Video and Video-to-Video Search System

Context

You are tasked with designing an end-to-end multimodal retrieval system that supports both:

  • Text-to-video search (user enters a query, returns the most relevant video segments)
  • Video-to-video search (user provides a reference video, returns visually/semantically similar segments)

Assume a catalog that grows to billions of video clips/segments, with global users and strict latency, cost, and safety requirements.

Requirements

  1. Model architecture
    • Encoders for text and video frames/clips (e.g., dual encoders)
    • Temporal aggregation for video (frame-to-clip, clip-to-video)
    • Shared embedding space, similarity function, and scoring
    • Pretraining/fine-tuning objectives (contrastive, multi-task), negative sampling strategies
    • Handling long videos (clip sampling, segment-level indexing)
    • Optional cross-encoder re-ranking for top-K
  2. Infrastructure
    • Offline/online feature extraction pipelines
    • Ingestion and metadata enrichment (e.g., ASR, OCR, tags)
    • Indexing in a vector store (HNSW/IVF, PQ/OPQ, sharding, replication)
    • Storage layout for thumbnails and short preview clips
    • Query routing and aggregation across shards
    • Latency and cost targets; caching strategies
    • Scalability to billions of clips/segments
  3. Monitoring, safety, and evaluation
    • System and model monitoring (SLOs, drift, recall)
    • Abuse/content safety (NSFW, spam, PII)
    • Evaluation via offline metrics and online A/B tests
    • Relevance feedback and incremental model updates

Clarifying Questions to Ask Guidance

  • Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
  • State explicit assumptions before making sizing or architecture decisions.
  • Prioritize the functional path first, then address reliability, security, observability, and rollout.

What a Strong Answer Covers Guidance

  • A scoped requirements summary with concrete non-goals and success metrics.
  • ML-specific data, model, evaluation, serving, and monitoring choices.
  • Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
  • A validation, monitoring, migration, and launch plan appropriate for the risk level.

Follow-up Questions Guidance

  • What breaks first at 10x traffic or data volume?
  • How would you degrade gracefully during dependency failures?
  • What metrics and alerts would prove the design is healthy after launch?

Browse More Questions

More ML System Design•More Runway•More Machine Learning Engineer•Runway Machine Learning Engineer•Runway ML System Design•Machine Learning Engineer ML System Design