Explain the Server-Side Stack of an AI App: Orchestration, Retrieval and Model Serving

As a product manager candidate, you are asked to explain how an AI product works behind the scenes. Choose a product, name its main backend components, and trace what happens to a user's request from the moment it leaves the app until the answer comes back. ```hint Follow one request, then look offline Pick one concrete AI feature and follow a single request from the user's screen to the model and back. Then ask what has to happen before any request arrives so the model has the right data to work with. ``` ### Clarifying Questions - Is the product built on a hosted foundation model from a provider, or does the company train and serve its own models? - Is it an interactive assistant that answers in real time, or a system that scores data in the background, such as scanning files or logs? - Does it answer from each customer's own documents, and must one customer's data stay isolated from every other customer's? - How deep should the explanation go: component-level boxes for a product audience, or named technologies for each layer? ### What a Strong Answer Covers - A concrete product choice and the requirements that shape its backend: latency, scale, model sourcing and data sensitivity. - The online request path from client and gateway through orchestration, retrieval and model inference back to the user, including streaming. - The background paths: ingesting and indexing knowledge, long-running jobs, evaluation and model updates. - Storage choices, and how the system scales, degrades under failure and keeps inference cost under control. - AI-specific risks such as prompt injection, leakage between customers and hallucination, and how quality is measured in production. ### Follow-up Questions - Cost per request doubles a month after launch. Which components do you inspect first, and which levers can bring it down? - How would the architecture change if a large enterprise customer requires that its data never leave its own cloud account? - A document a customer uploaded contains hidden instructions that make the assistant reveal content from another document. Where in your architecture do you stop that?