# Improve an ML Inference Service Review an existing ML inference-service design and propose concrete corrections around dynamic batching, load shedding, rate limiting, caching, and idempotency. Explain how the policies interact under overload and how you would prove that the revised service meets latency and correctness goals. ### Constraints & Assumptions - Requests have different tenants, priorities, payload sizes, and latency budgets. - Model execution benefits from batching but waiting for a batch adds latency. - Retries and duplicate client requests occur. - Capacity is finite and overload must not cause unbounded queues. - Cached results are safe only for fully identified deterministic inputs and model versions. ### Clarifying Questions to Ask - What are target latency percentiles, throughput, and error budgets? - Is inference deterministic and are personalized inputs cacheable? - Which requests may be dropped, delayed, or degraded? - What GPU memory and batch-size limits apply? ### Part 1 - Admission and overload Design tenant and global rate limits, bounded queues, deadlines, prioritization, and load shedding. #### What This Part Should Cover - Admission before scarce work is consumed - Fair quotas and burst handling - Deadline-aware queueing - Explicit retryable versus terminal rejection ### Part 2 - Batching and execution Explain compatible-request grouping, maximum batch size, maximum wait, worker scheduling, and protection against head-of-line blocking. #### What This Part Should Cover - Shape/model/version compatibility - Latency-throughput control loop - Cancellation and expired request removal - Batch-level versus item-level failures ### Part 3 - Caching and idempotency Define safe cache and request keys, result versioning, retry behavior, and invalidation. #### What This Part Should Cover - Full semantic input and model digest in keys - Separation of request deduplication from result caching - Privacy and tenant boundaries - Metrics proving hits are correct ```hint Budget queue time explicitly Subtract preprocessing and model-execution estimates from the request deadline; that remaining budget should constrain batch wait and queue admission. ``` ### What a Strong Answer Covers - Coordinated admission, bounded queues, and defensible shedding - Dynamic batching that respects compatibility and deadlines - Correctly scoped idempotency and cache identities - Feedback metrics, failure isolation, and overload tests ### Follow-up Questions 1. How should batch size change when p99 latency is healthy but GPU utilization is low? 2. Can an idempotency key safely return a result after the model version changes? 3. How would a long input be prevented from delaying many short inputs?