Design Large-Scale Inference Serving

Quick Overview

This question evaluates understanding of large-scale ML inference systems, assessing competencies in capacity planning, latency and tail-latency engineering, memory and bandwidth estimation, hardware selection (CPUs/GPUs/specialized accelerators), batching and caching trade-offs, and reliability concerns such as out-of-memory prevention and recovery. It is commonly asked to test practical system-design skills for production deployment by requiring back-of-the-envelope QPS and resource estimates and reasoning about operational trade-offs; this belongs to the ML system design category and emphasizes practical application over purely conceptual theory.

Design Large-Scale Inference Serving

Company: Waymo

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: medium

Interview Round: Technical Screen

Design a production inference serving system for a machine learning model used by 100 million daily active users. Your answer should cover: traffic assumptions and back-of-the-envelope QPS estimates; memory requirements for model weights, activations, caches, and batching; network and accelerator bandwidth estimates; how to choose CPUs, GPUs, or specialized accelerators; how to optimize latency and tail latency; and how to prevent or recover from out-of-memory failures.

Overview: This question evaluates understanding of large-scale ML inference systems, assessing competencies in capacity planning, latency and tail-latency engineering, memory and bandwidth estimation, hardware selection (CPUs/GPUs/specialized accelerators), batching and caching trade-offs, and reliability concerns such as out-of-memory prevention and recovery. It is commonly asked to test practical system-design skills for production deployment by requiring back-of-the-envelope QPS and resource estimates and reasoning about operational trade-offs; this belongs to the ML system design category and emphasizes practical application over purely conceptual theory.

|Home/ML System Design/Waymo
Waymo logo
Waymo
Nov 27, 2025
mediumMachine Learning EngineerTechnical ScreenML System Design
11
0

Design a production inference serving system for a machine learning model used by 100 million daily active users. Your answer should cover: traffic assumptions and back-of-the-envelope QPS estimates; memory requirements for model weights, activations, caches, and batching; network and accelerator bandwidth estimates; how to choose CPUs, GPUs, or specialized accelerators; how to optimize latency and tail latency; and how to prevent or recover from out-of-memory failures.

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More ML System Design•More Waymo•More Machine Learning Engineer•Waymo Machine Learning Engineer•Waymo ML System Design•Machine Learning Engineer ML System Design