Model Serving Interview Questions: Batching, GPUs, Latency, Autoscaling, and Rollbacks

Model serving interview questions test whether you can turn a trained model into a fast, reliable, and reversible production service. A strong answer connects batching, GPU capacity, latency targets, autoscaling, and rollback policy instead of treating them as separate buzzwords. Start with the request path and service-level objective, identify the bottleneck with measurements, and make every optimization preserve a safe failure mode.

If you want to rehearse with company-tagged prompts, explore PracHub ML system design interview questions before reading model answers. Attempt one prompt under a 45-minute limit, then use this guide to pressure-test the serving layer of your design.

Model serving interview questions with batching GPU latency autoscaling and rollback

What interviewers are actually testing

Model serving is the production layer that accepts prediction requests, schedules model execution, and returns outputs while meeting latency, availability, quality, and cost targets. Interviewers are testing systems judgment: can you turn a product requirement into an operable service and justify each choice with a measurement?

Interview signal	Weak answer	Strong answer
Requirements	“Use GPUs because inference is expensive.”	Defines traffic shape, model size, latency SLO, availability, output mode, and cost constraints.
Batching	Picks a large batch for throughput.	Derives a batching window from the latency budget and handles incompatible request shapes.
Scaling	Scales on CPU or GPU utilization alone.	Combines queue delay, outstanding work, token rate, saturation, and startup time.
Reliability	Adds retries everywhere.	Uses deadlines, admission control, bounded retries, idempotency, and a safe fallback.
Deployment	Says “use canary.”	Defines traffic stages, guardrails, rollback triggers, and version compatibility.

Start with the request path, not the GPU

Clarify the workload

Ask whether the service handles online predictions, asynchronous batch jobs, or both. A fraud score may need a sub-100 ms response, while a document-embedding job may tolerate minutes. Autoregressive generation also differs from fixed-cost classification because prompt length, output length, and key-value cache usage vary by request.

Clarify peak request rate, burstiness, payload size, model versions, tenant isolation, streaming, and availability. Then state the SLO. Without those facts, “optimize latency” and “maximize GPU utilization” are conflicting slogans.

Walk one request end to end

A useful baseline path is: gateway → authentication and quota → request validation → model/version router → admission queue → batch scheduler → model worker → post-processing → response. The worker loads an immutable model artifact and exposes readiness only after weights, runtime kernels, and required caches are warm.

Give each stage a portion of the latency budget so network, feature retrieval, queueing, inference, and serialization cannot hide behind one end-to-end number.

Batching interview questions

Why does batching help, and what does it cost?

GPUs often process a group of compatible requests more efficiently than isolated requests because the batch amortizes fixed work such as kernel launches and reading model weights. NVIDIA Triton calls this dynamic batching: requests are combined at serving time, up to configured batch and queue-delay limits.

The cost is waiting. A request that arrived first may sit in the queue while the scheduler searches for a fuller batch. Therefore, the best batch is not the largest batch the GPU can hold; it is the largest compatible batch that still respects the oldest request's remaining deadline.

A strong policy flushes on target batch size, maximum queue delay, or the earliest request deadline. Bucket by model version and compatible shape; for text generation, length-aware scheduling reduces padding waste and protects short requests.

Dynamic batching vs continuous batching

Dynamic batching works well for stateless, fixed-cost inference. Autoregressive generation needs a finer scheduler because requests finish after different numbers of decode steps. Continuous or inflight batching releases finished sequences and admits new ones between iterations, improving slot reuse.

Explain that capacity for an LLM worker is often constrained by GPU memory for weights and KV cache, not simply the number of requests. Admission should estimate tokens and cache footprint. Cancellation must also propagate when a client disconnects so abandoned generations stop consuming scarce decode capacity.

How do you prevent head-of-line blocking?

Use bounded queues, shape buckets, per-tenant quotas, and fair scheduling. When predicted queue delay exceeds the deadline, reject early or route to a degraded path instead of accepting work that will time out after consuming GPU capacity.

GPU capacity and scheduling questions

What belongs in the capacity model?

Start with measured throughput and latency for a specific model, precision, sequence-length mix, batch policy, and accelerator type. Then estimate replicas from peak work divided by safe per-replica capacity, adding headroom for failures, variance, and rollouts. Avoid quoting a universal “good GPU utilization” target because a latency-sensitive service may intentionally leave headroom.

Track memory separately from compute. GPU memory holds weights, runtime workspace, activations, and possibly KV cache. A worker can show modest compute utilization while rejecting work because memory is exhausted; high utilization can be healthy for batch inference and dangerous for a strict online SLO.

One model instance or several?

Multiple model instances may overlap transfer and computation, but they also compete for memory. Benchmark batching and instance count together: an extra instance can add latency without improving throughput once the GPU is saturated. Discuss tensor or pipeline parallelism only when one accelerator cannot hold the model or meet measured demand.

What would you measure?

Measure queue depth and age, batch size, inference duration, end-to-end p50/p95/p99, errors, GPU compute and memory, model-load time, and cost per success. For LLMs, add time to first token, inter-token latency, and tokens per second.

Model serving control loop from requests through GPU workers to autoscaling and rollback

Latency questions: diagnose before optimizing

Why are averages misleading?

An average can hide a small but painful group of slow requests. Google SRE guidance recommends percentile-capable monitoring because p50, p95, and p99 reveal how much of the request population violates the expected experience. Break end-to-end latency into queueing, preprocessing, inference, and post-processing so one chart does not hide the cause.

If p99 rises while GPU utilization is low, investigate traffic fragmentation, batch incompatibility, cold workers, CPU preprocessing, feature-store calls, synchronization, or a blocked queue. “Add GPUs” is weak because the accelerator may not be the bottleneck.

How do latency and throughput interact?

Increasing the batching window improves throughput but adds queue latency. More replicas may reduce queueing while producing smaller batches and higher cost. Use representative load tests and report a latency-throughput curve, including startup time; a new pod is not capacity until the model is ready.

Autoscaling interview questions

Which signal should drive scaling?

Kubernetes HPA can scale workloads from resource or custom metrics, but the metric must predict SLO risk. CPU utilization may describe preprocessing, while GPU utilization is a lagging and sometimes ambiguous signal. Queue depth, oldest-request age, outstanding tokens, concurrency, and predicted drain time often describe serving pressure more directly.

Scale out quickly enough to absorb a burst, but scale in slowly to prevent flapping and preserve warm capacity. Account for provisioning and model-load delay. If a GPU worker needs several minutes to become ready, scaling only after the queue is already deep guarantees an incident.

Should the service scale to zero?

Scale-to-zero can reduce cost for development or infrequently used models, but the first request pays the cold-start cost. KServe documents both the cost benefit and the latency trade-off. For a strict production SLO, keep minimum warm replicas, preload likely model versions, or route cold traffic to a smaller fallback.

How do you handle overload?

Autoscaling is not an overload strategy because new accelerators may be unavailable or slow to provision. Add admission limits, tenant quotas, bounded queues, deadlines, and graceful degradation. Preserve enough capacity for health checks and high-priority traffic, and shed work before cascading retries amplify the load.

Rollout and rollback questions

Shadow, canary, and A/B are different tools

Shadow traffic sends a copy of production requests to a new version without using its response. It validates compatibility, latency, and resource usage, but it does not prove user impact. A canary serves a small fraction of live traffic to limit blast radius. An A/B test is designed to estimate product impact and usually needs a stable assignment unit.

Version the model, runtime, preprocessing, feature schema, and routing policy together. Keep the last known-good release and enough warm capacity to reverse traffic quickly.

What should trigger rollback?

Define guardrails before rollout: error rate, p99 latency, timeout rate, GPU out-of-memory events, output-schema failures, safety violations, and quality or business metrics when labels arrive quickly enough. KServe's canary model illustrates the core mechanism: run versions side by side, shift weighted traffic, monitor, and return traffic to the known-good version when a step fails.

Rollback should be an operational command, not a meeting. Stop the rollout automatically for clear infrastructure regressions; require human review when the signal is noisy or the business metric has delayed feedback. Preserve request and model-version logs so the team can compare cohorts and reproduce failures.

A worked interview scenario

Suppose a recommendation ranker has a 200 ms p99 target and a 10x evening spike. Clarify candidate count, feature dependencies, model size, peak QPS, and fallback quality. Benchmark representative inputs across batch sizes, keep warm replicas, and scale on queue age plus outstanding work. Under overload, reduce the candidate set or use the previous lightweight ranker rather than waiting for timeouts.

Shadow a new version, then canary it with latency, error, and output-validity guardrails. If a predeclared threshold is breached, return traffic to the warm previous version and investigate with versioned traces.

This answer is strong because every component follows from a requirement, every optimization has a measurement, and every risky change is reversible.

Practice with PracHub questions

These PracHub question-bank records are practice material, not predictions of your exact interview. Each complete title in the first column opens the question and written solution.

PracHub question	Practice focus	Why it helps
Review an inference API design for scale	SLOs, multi-tenancy, capacity, rollback	Trains a senior review that connects operational choices instead of listing components.
Design GPU inference request batching	Deadline-aware batching and fairness	Forces a precise batching policy and exposes head-of-line blocking.
Design a GPU inference API	Online inference, memory, versions, scaling	Covers the complete low-latency GPU request lifecycle.
Design a batch inference API	Async jobs, idempotency, queues, partial failure	Contrasts throughput-oriented batch serving with online latency constraints.
Design an LLM Inference Serving System	Continuous batching, KV cache, streaming	Adds the scheduling and memory trade-offs unique to generative inference.

A seven-day preparation plan

Day	Focus	Deliverable
Day 1	Requirements and SLOs	Write one page of clarifying questions and a latency budget.
Day 2	Batching	Compare no batching, dynamic batching, and continuous batching.
Day 3	GPUs	Build a capacity estimate with memory, throughput, and headroom.
Day 4	Observability	Draw a dashboard that separates queue, execution, and end-to-end latency.
Day 5	Autoscaling	Choose signals, cooldowns, warm capacity, and overload behavior.
Day 6	Rollouts	Define shadow, canary, promotion, and rollback gates.
Day 7	Mock interview	Complete one PracHub prompt in 45 minutes and rewrite the weakest section.

Common mistakes to avoid

Do not jump straight to a vendor or GPU count. Tools implement a design; they do not replace requirements, measurements, and failure semantics. Do not optimize utilization while queues grow and deadlines expire, or assume low utilization proves overprovisioning when burst headroom is intentional. Finally, treat the artifact, runtime, feature contract, configuration, and routing policy as one reversible release.

Frequently asked questions

What is the difference between model deployment and model serving?

Deployment places a model artifact and its runtime into an environment. Serving is the ongoing system that accepts requests, schedules execution, returns predictions, scales capacity, observes behavior, and manages versions. An interview may start with deployment but usually evaluates how the service behaves under load and failure.

What is the best autoscaling metric for model serving?

There is no universal single metric. Queue age or predicted drain time often reflects user-visible pressure better than CPU or GPU utilization alone. Combine demand, saturation, and readiness signals, then test whether the policy protects the latency SLO during realistic bursts.

Why can dynamic batching increase latency?

The scheduler may deliberately wait for compatible requests so the GPU processes a larger batch. That increases throughput but adds queue time. Limit the wait with a deadline-aware flush policy and measure end-to-end percentiles rather than inference duration alone.

How should an LLM serving answer differ from a classifier serving answer?

LLM requests vary in prompt and output length, stream partial results, and consume KV-cache memory across many decode steps. Discuss continuous batching, token-aware admission, cancellation, time to first token, inter-token latency, and memory pressure in addition to standard request routing and scaling.

What makes a rollback safe for an ML service?

A safe rollback restores a compatible set of model weights, runtime, preprocessing, feature schema, and configuration. Keep the previous version deployable or warm, define rollback thresholds before the canary, preserve versioned logs, and verify that the fallback still meets the API contract.

Build an answer that survives follow-ups

Model serving interviews become much easier when you keep one causal chain visible: SLO → capacity → batching and scheduling → observability → autoscaling → rollout and rollback. If the interviewer changes a constraint, trace its effect through that chain instead of adding another box to the diagram.

Use the PracHub ML system design question bank to practice that reasoning against concrete prompts. Attempt the design first, compare it with the written solution, and turn every missed trade-off into the next mock interview's checklist.

Model Serving Interview Questions: Batching, GPUs, Latency, Autoscaling, and Rollbacks