Deploying a text-to-text generation system

Trace a conversational request through validation, retrieval, context construction, model serving, and output checks. Estimate storage and traffic under explicit assumptions, then identify the measurements needed to size the serving fleet.

The preceding lesson described training and evaluation infrastructure; it did not produce or certify a trained model. For this design exercise, assume an evaluated candidate model is available. Providers such as AWS, GCP, and Microsoft Azure can host parts of the service, but the following design examines the responsibilities and interfaces independently of a particular platform.

Begin with resource arithmetic, then connect the components into a request path. Treat the architecture as a proposal to test against the requirements, not evidence that those requirements have already been met.

The BOTECs chapter separates training and deployment estimates. Here, count:

• Storage estimation

• Inference servers

• Network bandwidth

Use decimal units and the hypothetical workload of 100 million daily active users making ten requests each per day. Assume separately that the service stores 100 million user profiles.

Storage estimation