Optimize Model Serving Under 200ms
Company: Xometry
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: medium
Interview Round: Technical Screen
A data science team gives you a trained model and asks you to deploy it as an online inference service. The requirement is that a single prediction must complete within 200 milliseconds. Describe how you would clarify the requirement, measure the baseline, optimize the model and serving stack, choose hardware, validate accuracy-latency tradeoffs, and monitor the system after launch.
Quick Answer: This question evaluates competency in deploying and optimizing machine learning models for low-latency online inference, covering model serving, latency profiling, hardware considerations, and managing accuracy–latency trade-offs within a 200ms SLO in the ML System Design domain.