Design GPU inference request batching

Q: Design GPU inference request batching

This question evaluates understanding of GPU inference batching, request queuing and routing, scheduling and autoscaling, throughput–latency trade-offs, multi-model/version management, failure handling, and observability within machine learning serving systems, and is in the ML System Design domain.

Q: How do I approach ML System Design interview questions?

ML System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master ml system design interviews.

Question

Design a system that serves online model-inference requests on GPUs. Requests arrive one at a time from clients, but GPU throughput is much better when compatible requests are grouped into batches.

Discuss how you would design a service that:

accepts low-latency inference requests,
batches compatible requests together,
routes work to GPU workers,
supports multiple models or model versions,
balances throughput against latency SLOs,
handles overload, failures, and observability.

Your design should cover the API, queueing model, batching strategy, scheduling policy, worker lifecycle, autoscaling signals, and the main trade-offs.

Design GPU inference request batching

Quick Overview

Solution

Comments (0)