Design a real-time model inference platform: deployment, scaling and monitoring

Design a platform for real-time model inference: teams hand it trained models, and applications call it for predictions inside their own request path. The discussion covered how models are deployed onto the platform, how it scales, and how it is monitored. ```hint What a team hands over Decide what a model version consists of besides its weights, and how a new version reaches production traffic without a single all-at-once switch. ``` ```hint Scale each model on the right signal Models differ in size, hardware and traffic. Think about what limits the throughput of one replica, what signal should add replicas, and what happens while a new replica is still loading. ``` ```hint Healthy servers, wrong answers A model can keep returning fast, valid responses while its predictions get worse. Decide what you can watch before the true outcomes are known. ``` ### Constraints and Clarifications - Not given: the number of models, request rates, latency targets, model types or hardware. Ask, or state your assumptions. ### Clarifying Questions - How many models and teams will use the platform, and how different are the models (small tree ensembles versus large neural networks that need GPUs)? - What latency budget do callers have, and is the API synchronous request and response? - Where do model inputs come from: the caller's request, lookups in a feature store, or both? - What does traffic look like: steady, daily peaks, or sudden spikes? - How fast must a bad model version be rolled back, and who approves a promotion to full traffic? - Do some predictions combine several models, such as an ensemble or a cascade? ### What a Strong Answer Covers - Settles the model mix, latency budget, traffic shape and input sources before designing - A clear split between the control plane (model registry, deployment controller) and the data plane (gateway, feature retrieval, model servers) - A safe deployment path: validation, shadow traffic, canary, automated rollback, versioned routing - Scaling that fits each model: the autoscaling signal, request batching, hardware choice, cold starts, packing many small models, isolation between teams - Monitoring at three levels: infrastructure, service (latency and errors per model version) and model quality (input drift, prediction distribution, accuracy once outcomes arrive) ### Follow-up Questions - A new model version looks fine offline, but a business metric drops after rollout. How should the platform have caught it, and how quickly can it roll back? - One model needs a GPU and receives bursty traffic. How do you meet its latency budget without paying for idle GPUs? - Hundreds of rarely called models must all be servable. How do you avoid one always-on replica per model? - Feature lookups take most of a request's latency. What can the platform do about it?