Choose Between Two Machines and Allocate Work from Performance Curves

# Choose Between Two Machines and Allocate Work from Performance Curves You are given two machine types with different specifications and measured latency-versus-throughput curves for the same workload. Explain when each machine is faster, when each is more efficient, and how you would distribute incoming work across them while meeting an agreed latency objective. The curves may change with batch size, concurrency, queue depth, and request shape. Base the decision on measured operating regions rather than a single peak-throughput number. ### Constraints & Assumptions - Both machines produce equivalent results for the workload. - Per-request work may vary, and batching may delay an individual request while improving throughput. - Queueing time counts toward end-to-end latency. - Exact arrival distribution, service objective, cost, failure model, and whether work can be split should be clarified. ### Clarifying Questions to Ask - Is the objective minimum latency, maximum throughput, cost per result, or a constrained combination? - Which latency percentile and traffic burst define the service objective? - Can requests be classified by shape or estimated cost before routing? - May one request be divided across machines, or may only independent requests be distributed? - How quickly can capacity scale, and what warm-up or model-load cost applies? ### What a Strong Answer Covers - Identification of curve crossover points and feasible operating regions under the required latency percentile. - Separation of service time, batching delay, queueing delay, throughput, utilization, and cost. - Headroom rather than operation at the unstable saturation edge. - Routing by request class or deadline when machines favor different shapes, plus weighted load balancing when they are interchangeable. - Admission control, queue limits, spillover, retries, and behavior when one pool fails or its curve drifts. - A workload-allocation method validated through replay or load tests using the real arrival distribution. - Monitoring and periodic recalibration instead of treating benchmark curves as permanent specifications. ### Follow-up Questions 1. Machine B has higher peak throughput but worse low-load latency. Which requests should it receive? 2. Why can sending work to the machine with the shortest current queue still produce a poor allocation? 3. How much headroom would you retain near the point where the latency curve bends sharply? 4. What changes if requests have two distinct size classes that reverse the machines' ranking? 5. How would you react when live latency no longer matches the benchmark curve?