Design an internal ML platform that lets data scientists train models, register reproducible versions, and deploy them for both online and batch inference. The platform should separate control-plane workflows from data-plane execution. Discuss how training code, data versions, model artifacts, configuration, and lineage move through the system and how a deployment is promoted or rolled back. ### Constraints & Assumptions - Training jobs can require different compute resources and may run for a long time. - Online inference prioritizes low latency; batch inference prioritizes throughput and resumability. - A model version must be traceable to code, data, configuration, and evaluation evidence. ### Clarifying Questions to Ask - Which frameworks and compute environments must be supported? - What online latency and availability targets exist? - Who approves a model for production, and what automated gates apply? ```hint Give online and batch separate execution paths They can share the registry and feature definitions while using different scheduling, scaling, and output mechanisms. ``` ### What a Strong Answer Covers - Job submission, resource scheduling, experiment metadata, artifact storage, and lineage. - A model registry with immutable versions, evaluation gates, staged rollout, and rollback. - Online serving, batch inference, feature consistency, observability, and security boundaries. ### Follow-up Questions - How would you prevent training-serving feature skew? - How would an online endpoint recover from a bad model release? - How would you make a failed week-long batch job resumable?