# Design a Synchronous Text Classification API A trained machine-learning model classifies text. Design the system that exposes this model through synchronous API endpoints for customers. Cover request flow, load balancing, throttling, failure handling, stateless serving, model deployment, and how the design can expand as traffic and model count grow. A database is not inherently required for the inference path. Introduce persistent storage only when a stated requirement needs it, and explain why. ### Constraints & Assumptions - Callers expect a classification response in the request/response cycle. - Requests can be retried and may arrive in bursts. - Model artifacts are immutable and versioned. - Individual serving instances can fail at any time. - The prompt does not provide latency, throughput, payload-size, or availability numbers; identify them before sizing. ### Clarifying Questions to Ask - What are the latency percentile and availability objectives? - What are expected and peak request rates, text sizes, and number of tenants? - Are CPU, GPU, or other accelerators required, and can requests be batched? - Must callers select a model version, and is explanation or confidence data returned? - What privacy, retention, abuse-prevention, and regional requirements apply? ### Solving Hints - Keep the online path small and make model loading a deployment concern rather than a per-request action. - Distinguish overload from dependency failure and define a bounded response for each. - Explain how a model version can be rolled out and rolled back without mixing untraceable results. ### What a Strong Answer Covers - API contracts, authentication, validation, request limits, and tenant-aware rate limiting. - Stateless gateways and inference workers behind health-aware load balancing. - Model artifact distribution, warm-up, version routing, canaries, and rollback. - Bounded queues, load shedding, timeouts, retries only where safe, and circuit breaking. - Autoscaling and partitioning choices based on measured resource cost. - Observability for service and model behavior without retaining sensitive text unnecessarily. - A credible evolution path for multiple models, regions, and hardware classes. ### Follow-up Questions 1. How would you avoid a cold-start latency spike during a model rollout? 2. What should the API return when every healthy worker is at capacity? 3. When would dynamic batching help, and when would it violate the synchronous latency objective? 4. How would you diagnose a quality regression that affects only one model version?