Design an Offline Inference Platform with Hardware-Aware Routing
Company: Gimlet Labs
Role: Member of Technical Staff, ML Systems
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
## Interview Prompt
Design an offline large-model inference platform that can place different phases
of work on different hardware pools, including older accelerators suitable for
decode. Focus on job ingestion, batching, model and data placement, routing,
prefill/decode coordination, retries, capacity, fairness, and how the router uses
measured hardware characteristics rather than static assumptions.
### Constraints & Assumptions
- Jobs are offline but have explicit completion deadlines and tenant priorities.
- Hardware pools differ in memory, bandwidth, compute, and failure rate.
- Intermediate model state can be expensive to transfer between phases.
- One tenant cannot monopolize the fleet with very large batches.
### Clarifying Questions to Ask
- Which model families, sequence lengths, and quantization formats are supported?
- Are prefill and decode always separated, optionally separated, or colocated by default?
- What throughput, deadline, cost, and quality objectives drive routing?
### What a Strong Answer Covers
- A durable job model, admission control, tenant quotas, and deadline-aware scheduling.
- Profiling-based compatibility and cost models for each model, phase, and hardware pool.
- Routing that accounts for queueing, KV-cache transfer, memory pressure, and batch efficiency.
- Idempotent phase transitions, checkpointing, retry placement, and straggler handling.
- Metrics and experiments for tokens per second, deadline misses, utilization, transfer cost, and quality drift.
### Follow-up Questions
- When should the router colocate prefill and decode despite lower device utilization?
- How would you prevent stale performance profiles from causing a routing collapse?
- What happens when a hardware pool disappears mid-job?
Overview: Design an offline large-model inference platform that routes prefill and decode across profiled hardware pools while balancing deadlines, fairness, batching, transfer cost, retries, and utilization.