System Design: From AI Prototype to Production

Context

Assume you are designing a user-facing AI-powered feature for a web/mobile product. Some decisions must return predictions in real time (online inference), while others can be scheduled periodically (batch inference). Propose a production-grade design that addresses reliability, safety, privacy, and continuous improvement.

What to Cover

1. Deployment architecture
   • Online vs. batch inference
   • Service boundaries (feature service/store, inference service, retrieval/ranking, data pipelines)
   • Scaling, availability, and failure isolation
2. Model and feature versioning
   • Versioning for models, datasets, features, and transformations
   • Reproducibility and lineage
3. CI/CD for models (MLOps)
   • Training/validation pipeline, automated tests, promotion gates
   • Packaging and environment management
4. Monitoring and observability
   • Latency/throughput/error SLI/SLOs
   • Data/feature drift, training-serving skew, model quality
   • Cost/resource monitoring
5. Rollouts and safety
   • Shadow, canary, blue/green, and A/B testing
   • Guardrails and rollback procedures
6. Data and feedback instrumentation
   • What to log at inference and training time
   • Labeling strategy (in-product feedback, delayed outcome labels, active learning)
   • Consent, privacy, compliance, and data retention

Be explicit about trade-offs and provide practical guardrails and fallbacks.