Design an Embedding and Classification Inference API

Design an API service for embeddings and classification. The report gives only this one-line prompt, so settle the scope with the interviewer first. In its usual form, clients send content such as text; the service runs an embedding model that turns each item into a vector, runs a classifier that assigns labels to the item from that vector, and returns the vector, the labels, or both. ```hint The expensive step is shared The embedding is the costly model call, and the classifier consumes its output. Decide where it is computed, whether it is kept, and what a repeated request for the same content should cost. ``` ```hint Accelerators want batches, callers want latency Decide how individual requests reach the GPU workers so that the hardware is well used without making any single caller wait too long, and what happens when requests arrive faster than the workers can serve them. ``` ### Constraints and Clarifications - Not given: input types and sizes, model sizes, request volume, latency targets, the label set, and whether callers need the raw vectors. Ask, or state your assumptions. ### Clarifying Questions - What inputs: text only, images, or both? What is the size limit per item? - Do callers need the embedding vector, the labels, or both? Is the label set fixed, or can it change often? - Synchronous low-latency calls, large offline batches, or both? What latency target and request volume? - Do the embedding model and the classifier change over time, and must results from different model versions stay comparable? - Is the service multi-tenant with quotas, and are there retention rules for the content and the vectors? ### What a Strong Answer Covers - Scopes inputs, outputs, synchronous and batch modes, and latency and throughput targets, and sizes GPU capacity from them - A clear API: a synchronous endpoint, an asynchronous batch-job interface, model versions in every response, and per-item errors - A serving path that computes each embedding once and feeds it to the classifier, with dynamic batching on GPU workers and admission control - Caching or deduplication of results keyed by content and model version, and a policy for storing vectors - Model versioning and rollout, consistency between embedding and classifier versions, and re-embedding when the embedding model changes - Autoscaling, backpressure, failure handling, and monitoring of latency, GPU use and prediction drift ### Follow-up Questions - A new embedding model ships, and vectors from the old model are not comparable with the new ones. How do you migrate callers and stored vectors? - One tenant submits a very large batch job. How do you keep interactive latency stable for everyone else? - New labels must be added every week without retraining the embedding model. How does your design support that? - How do you detect that classification quality is degrading in production when live traffic has no labels?