Design an Autoscaling LLM Inference Serving System

Quick Overview

Design an LLM inference serving system that adds and removes model replicas automatically as traffic changes while meeting latency targets and controlling GPU cost. It tests choosing scaling signals, handling slow cold starts, load-aware routing, safe scale-down, multi-model serving and graceful behavior under overload.

Design an Autoscaling LLM Inference Serving System

Company: Crusoe

Role: Software Engineer

Category: ML System Design

Difficulty: medium

Interview Round: Onsite

Design an inference serving system for large language models that scales automatically: it adds model replicas when demand rises and removes them when demand falls, while meeting latency targets and keeping GPU cost under control. The report gives only this one-line prompt, so the models, scale and latency targets are open: settle them with the interviewer before designing. ```hint Pick the signal GPU utilization and CPU load are poor triggers here. Think about which measurements show that requests are about to miss their latency targets before they actually do. ``` ```hint Scaling up is slow Estimate how long a new replica takes from the decision to scale until it serves its first request, and design around that delay. ``` ### Clarifying Questions - How many models, of what sizes, and does one replica need one GPU or several? - What does traffic look like: steady, daily cycles, sudden spikes? Interactive chat, batch jobs, or both? - What are the latency targets for time to first token and time per output token, and at which percentile? - Is GPU capacity effectively unlimited from a cloud provider, or a fixed pool shared with other workloads? - Are prefill and decode served on the same replicas or disaggregated into separate pools? ### What a Strong Answer Covers - Requirements, latency targets and rough GPU-count estimates agreed first - Routing and load balancing that account for queue length, prompt size and KV cache, not only round robin - Scaling signals tied to the latency targets (queue time, KV cache usage, token throughput), with hysteresis and cooldowns - Handling the slow cold start: weight loading, warm capacity, predictive scaling, and admission control while capacity arrives - Safe scale-down that drains long-running streaming requests - Multi-model serving, GPU capacity limits, failure handling, cost and observability ### Follow-up Questions - Traffic triples within a minute, but new replicas need several minutes to start. What happens to requests in the meantime? - How would you scale prefill and decode servers independently in a disaggregated deployment? - How do you serve a long tail of rarely used models without keeping a GPU warm for each? - How do you prevent oscillation, where the system keeps adding and removing replicas?

Overview: Design an LLM inference serving system that adds and removes model replicas automatically as traffic changes while meeting latency targets and controlling GPU cost. It tests choosing scaling signals, handling slow cold starts, load-aware routing, safe scale-down, multi-model serving and graceful behavior under overload.

Read the full Crusoe Software Engineer interview experience this question came from

Crusoe logo
Crusoe
Nov 2, 2026
mediumSoftware EngineerOnsiteML System Design
0
0

Design an inference serving system for large language models that scales automatically: it adds model replicas when demand rises and removes them when demand falls, while meeting latency targets and keeping GPU cost under control. The report gives only this one-line prompt, so the models, scale and latency targets are open: settle them with the interviewer before designing.

Clarifying Questions Guidance

  • How many models, of what sizes, and does one replica need one GPU or several?
  • What does traffic look like: steady, daily cycles, sudden spikes? Interactive chat, batch jobs, or both?
  • What are the latency targets for time to first token and time per output token, and at which percentile?
  • Is GPU capacity effectively unlimited from a cloud provider, or a fixed pool shared with other workloads?
  • Are prefill and decode served on the same replicas or disaggregated into separate pools?

What a Strong Answer Covers Guidance

  • Requirements, latency targets and rough GPU-count estimates agreed first
  • Routing and load balancing that account for queue length, prompt size and KV cache, not only round robin
  • Scaling signals tied to the latency targets (queue time, KV cache usage, token throughput), with hysteresis and cooldowns
  • Handling the slow cold start: weight loading, warm capacity, predictive scaling, and admission control while capacity arrives
  • Safe scale-down that drains long-running streaming requests
  • Multi-model serving, GPU capacity limits, failure handling, cost and observability

Follow-up Questions Guidance

  • Traffic triples within a minute, but new replicas need several minutes to start. What happens to requests in the meantime?
  • How would you scale prefill and decode servers independently in a disaggregated deployment?
  • How do you serve a long tail of rarely used models without keeping a GPU warm for each?
  • How do you prevent oscillation, where the system keeps adding and removing replicas?