On-Device vs Server Inference: Choosing Where an App's ML Model Runs at Scale

You are building an app that will serve millions of users and relies on a machine learning model. Would you run the model on your servers or on users' devices? Explain how you would make the call. ```hint Let the feature decide Before picking a side, ask what the model does and what the user experiences while waiting for its output. Then check whether the user count really pushes you the way it first seems to. ``` ### Clarifying Questions - What does the model do, and how large is it: a small classifier, or a large generative model? - Must the feature work offline or in real time, for example on a live camera feed or while the user types? - How sensitive is the input: personal photos, voice or health data that users may not want uploaded? - Which devices must it support: recent high-end phones only, or older low-end phones and web browsers too? ### What a Strong Answer Covers - Grounding the choice in the use case rather than in the user count alone - The main trade-offs: latency, offline use, privacy, serving cost, model size and accuracy, battery, device fragmentation, and speed of updates - How scale affects each option: server cost that grows with usage versus compute spread across users' own devices - When a hybrid split makes sense, and which part of the work runs where - How the decision would be validated and monitored after launch ### Follow-up Questions - The model is retrained every week. Does that change your answer? - How would you keep a compressed on-device model close in quality to the full server version, and how would you notice if it drifted? - Would your answer change if the model were a large language model powering a chat feature?