Introduction to Generative AI System Design

Concept overview

Introduction to Generative AI System Design is about designing a product around model behavior, not just calling a model endpoint. Generative AI systems have the usual distributed-systems concerns, plus quality, context, safety, latency, cost, and evaluation problems that show up in different ways for each user request.

For a AI assistant platform, the core design question is where reliability comes from. It may come from retrieval, prompting, fine-tuning, tools, moderation, caching, or human review. The model is powerful, but the product becomes trustworthy only when the surrounding system constrains and measures it.

Use this lesson as a system-design chapter. The best answers explain the model path, the data path, the control path, and the feedback path.

tradeoff matrix

Introduction to Generative AI System Design design tradeoffs

Choice	When it works	Risk to watch
Prompting	Fast iteration and low setup cost	Quality may vary across user inputs
Retrieval	Answers depend on fresh private knowledge	Bad chunks or weak ranking cause wrong context
Fine-tuning	Style or narrow task behavior matters	Training data quality and evaluation become the bottleneck
Tool use	The model needs external actions or facts	Tool errors need guardrails and recovery

Mental model

Think of Introduction to Generative AI System Design as a controlled generation pipeline.

Input arrives from the user. The system prepares context, chooses a model or route, applies tools or retrieval when needed, checks safety and policy, returns output, and records signals for evaluation. Each stage can improve quality, but each stage also adds latency, cost, and failure modes.

The useful interview move is to separate model quality from product reliability. A bigger model may improve some outputs, but the system still needs fallbacks, guardrails, evaluation sets, observability, and cost controls.

Worked example: AI assistant platform

Suppose a product team ships a model-backed feature that must be useful, safe, and affordable. The naive design sends the user request to a model and returns the output. That may be enough for a demo, but it is not enough for an interview-quality system.

Start with the task boundary. What does the user expect the system to produce? What latency can they tolerate? What inputs are unsafe or unsupported? What output must be blocked, reviewed, or regenerated? These constraints decide whether you need retrieval, tools, fine-tuning, caching, streaming, moderation, or human review.

Then design the orchestration path. The system may rewrite the query, retrieve context, rank chunks, build a prompt, choose a model, call tools, run safety checks, and cache the result. Each step should have a reason. More steps can improve control, but they also create more places to fail.

Evaluation is part of the system. You need offline test sets for quality, automated checks for safety and format, human review for ambiguous cases, and online metrics for user value. Cost and latency are also evaluation metrics. A technically impressive design that is too slow or too expensive will not survive production.

Tradeoffs and failure modes

The main tradeoff in Introduction to Generative AI System Design is control versus flexibility. Generative models are useful because they handle messy input, but the same flexibility creates hallucination, prompt injection, unsafe output, inconsistent format, cost spikes, and hard-to-debug failures.

Common mitigations include retrieval boundaries, tool permissions, structured output checks, safety classifiers, caching, fallback models, rate limits, human review, and evaluation suites. The right answer depends on the product risk, not on a universal GenAI pattern.

Interview prep layer

What interviewers are testing

Interviewers are testing whether you can design the system around model behavior, context, safety, cost, latency, and evaluation. They are less interested in whether you memorized a framework and more interested in whether you can apply it cleanly when the prompt is ambiguous.

Short model answer

"For Introduction to Generative AI System Design, I would define the user task and quality bar, then choose the generation strategy: prompting, retrieval, fine-tuning, tool use, or a hybrid. I would design the orchestration path, guardrails, fallback behavior, evaluation, and cost controls."

Common traps

• Treating the model call as the whole system.

• Assuming a larger model removes the need for evaluation.

• Ignoring prompt injection, unsafe output, latency, or cost.

Practice drill

Take AI assistant platform and explain the lesson in three minutes. Spend the first minute on the concept, the second minute on the design or analysis choices, and the third minute on tradeoffs, risks, and what you would do next.

Quick self-check

Before moving on, answer these out loud:

• What context does the model need, and where does it come from?

• What quality, safety, latency, and cost metrics matter?

• What fallback should run when the model is uncertain or unsafe?

• How would you evaluate the system before and after launch?

Topic walkthrough

Why generative AI system design matters

If you have been following the AI industry over the past few years, you have witnessed something notable. Generative AI has moved from a niche research topic to the backbone of products used by hundreds of millions of people. ChatGPT reached 100 million users faster than any consumer application in history. Midjourney generates millions of images daily. ElevenLabs powers voice synthesis for content creators worldwide. Behind each of these products lies a carefully designed system that balances model quality, latency, cost, and reliability at massive scale.

This course exists because there is a growing gap between understanding how generative models work in theory and knowing how to design the systems that bring them to production. Whether you are preparing for a system design interview at a top AI company or building your own GenAI product, the ability to reason about these systems end-to-end is now a critical skill.

Interview tip: When an interviewer asks you to "design a system like ChatGPT," they are not asking you to derive the transformer architecture from scratch. They want to see that you can reason about the full stack: data pipelines, training infrastructure, serving architecture, evaluation, and deployment.

The intersection of distributed ML and system design

Traditional system design focuses on building scalable web services: load balancers, databases, caches, message queues, and CDNs. GenAI system design introduces an entirely new dimension: the machine learning model itself becomes the most expensive, most complex, and most latency-sensitive component in your architecture.

Consider the differences:

In GenAI system design, you need to think about both the traditional distributed systems concerns AND the ML-specific challenges. A well-designed GenAI system handles data collection, preprocessing, distributed training across hundreds of GPUs, model evaluation, efficient serving with tight latency budgets, and continuous monitoring of model quality in production.

What makes GenAI systems uniquely challenging

GPU memory is the new bottleneck. A single GPT-3 scale model (175B parameters) requires roughly 350GB just to store its weights in FP16. That is more than four A100 GPUs (80GB each) just for the weights, before you account for activations, KV cache, or optimizer states during training.

Inference cost scales with output length. Unlike a traditional API where serving cost is roughly constant per request, GenAI models have costs that scale with the number of tokens generated. A 500-token response costs roughly 5x more compute than a 100-token response.

Quality is probabilistic, not deterministic. A traditional web service either returns the correct data or it does not. A GenAI system might generate a plausible-sounding but factually incorrect response. Monitoring and evaluation require entirely different approaches.

Training is a massive distributed computing problem. Training a frontier model like GPT-4 or Llama 3 requires thousands of GPUs running in coordination for weeks or months. A single GPU failure can waste hours of work if checkpointing is not done properly.

Generative AI modalities

This course covers the major generative AI modalities that companies are building products around today. Each modality has its own unique challenges, but they share common architectural patterns.

Text-to-text generation

This is the most mature and widely deployed modality. Systems like ChatGPT, Claude, and Gemini take text prompts and generate text responses. The core architecture is the autoregressive transformer, which generates text one token at a time.

Key challenges: Managing long context windows (up to 128K+ tokens), reducing latency for streaming responses, handling multi-turn conversations, content safety filtering, and managing the enormous cost of serving billions of tokens per day.

Scale reference: OpenAI reportedly serves over 100 million weekly active users, processing billions of tokens daily across their API and ChatGPT products.

Text-to-image generation

Systems like DALL-E, Midjourney, and Stable Diffusion generate images from text descriptions. Most modern systems use diffusion models that start with random noise and iteratively denoise it into a coherent image, guided by the text prompt.

Key challenges: Generating high-resolution images (1024x1024+) with fine details, ensuring text-image alignment, controlling generation (style, composition, specific elements), NSFW filtering, and managing the compute cost of the iterative denoising process.

Scale reference: Midjourney has generated over a billion images since launch, with millions of new images created daily.

Text-to-speech generation

Systems like ElevenLabs and Play.ht convert text into natural-sounding speech. Modern TTS systems can clone voices from short audio samples and generate speech with natural prosody, emotion, and style.

Key challenges: Real-time streaming synthesis for conversational AI, voice cloning with minimal reference audio, maintaining speaker consistency across long passages, multilingual support, and preventing misuse (deepfake voices).

Scale reference: ElevenLabs processes millions of characters of text per day, serving content creators, game developers, and accessibility applications.

Text-to-video generation

The newest frontier, with systems like Sora and Runway Gen-3 generating video clips from text descriptions. This combines the challenges of image generation with temporal consistency and motion modeling.

Key challenges: Maintaining temporal consistency across frames, modeling realistic motion and physics, extremely high compute requirements (orders of magnitude more than image generation), long generation times, and video quality at higher resolutions.

Scale reference: Video generation is still in its early stages commercially, but the compute requirements are staggering. Generating a single minute of high-quality video can require hundreds of GPU-seconds.

The scaled framework

Throughout this course, we use a systematic 6-step framework called SCALED for approaching GenAI system design problems. This framework gives you a structured way to work through any GenAI design question, whether in an interview or in a real project.

Here is how you might use the SCALED framework to think through a text-to-text system:

S - Scenario: "We need a conversational AI assistant that handles 10M daily active users, supports multi-turn conversations up to 8K tokens, and responds in under 2 seconds for the first token."

C - Capabilities: "We will use a transformer-based autoregressive model with 70B parameters, trained on a mix of web data and curated instruction data, fine-tuned with RLHF for helpfulness and safety."

A - Architecture: "The system has an API gateway for rate limiting and auth, a conversation manager for context handling, a model serving layer with continuous batching using vLLM, and a safety filter pipeline."

L - Latency/Throughput: "Time-to-first-token target is 500ms. We need to serve 1,000 requests per second at peak. Each H100 can serve roughly 50 concurrent requests with continuous batching, so we need about 20 H100s for the model serving layer alone."

E - Evaluation: "We track perplexity on held-out data, human preference ratings via thumbs up/down, safety metrics from our content filter, and latency percentiles (p50, p95, p99)."

D - Deployment: "We use canary deployments for new model versions, rolling out to 1% of traffic first. Model weights are stored in S3 and loaded on instance startup. We use Kubernetes for orchestration with GPU node pools."

Interview tip: Memorize the SCALED acronym. In an interview, explicitly tell the interviewer which step you are on. This demonstrates structured thinking and makes it easy for them to follow your design.

Why companies ask GenAI system design questions

The explosive growth of generative AI has created a new category of interview questions. Companies like Google, Meta, OpenAI, Anthropic, Amazon, Microsoft, and many startups now ask candidates to design GenAI systems. Here is why:

The talent gap is real. There are far more companies wanting to build GenAI products than there are engineers who understand how to design these systems end-to-end. If you can demonstrate this knowledge, you stand out immediately.

It tests both ML and systems knowledge. GenAI system design sits at the intersection of machine learning and distributed systems. It tests whether you can reason about model architectures, training pipelines, serving infrastructure, and operational concerns all at once.

It reflects real work. At companies building GenAI products, engineers routinely make decisions about model selection, serving architecture, cost optimization, and quality monitoring. These interviews mirror the actual work.

What interviewers evaluate

When you are asked to design a GenAI system, interviewers are looking for:

• Breadth of knowledge: Can you reason about the full stack, from data to deployment?

• Depth on tradeoffs: Can you articulate why you would choose one approach over another?

• Quantitative reasoning: Can you estimate compute requirements, costs, and latency?

• Practical awareness: Do you understand real-world constraints like GPU availability, cost, and failure modes?

• Communication: Can you explain complex ML concepts clearly?

Key differences between traditional and GenAI system design

If you have prepared for traditional system design interviews (designing URL shorteners, chat systems, or news feeds), you already have a solid foundation. GenAI system design builds on those skills but adds several new dimensions.

You must think about two phases: training and serving

Traditional system design only deals with the serving phase. GenAI system design requires you to think about both:

Training phase: How do you collect and process training data? How do you distribute training across a GPU cluster? How long will training take? How much will it cost? How do you evaluate whether the model is good enough?

Serving phase: How do you load a massive model onto GPUs? How do you handle the sequential nature of autoregressive generation? How do you batch requests efficiently? How do you manage GPU memory?

Cost is dominated by compute, not storage

In traditional systems, storage (databases) is often the main cost driver. In GenAI systems, GPU compute dominates everything. A single H100 GPU often costs about $8-$12/hour on cloud providers. A cluster of 1,000 H100s can cost about $8,000-$12,000/hour, or roughly $190,000-$290,000/day. Serving costs scale with the number of tokens generated.

Latency has a different character

In traditional systems, latency is primarily network and I/O bound. In GenAI systems, latency is compute bound. Generating a 200-token response requires 200 sequential forward passes through the model. You cannot parallelize this. The only ways to reduce latency are to use faster hardware, smaller models, or techniques like speculative decoding.

Quality monitoring is fundamentally different

You cannot write unit tests for a generative model. The model might generate a perfectly grammatical, confident-sounding response that is completely wrong. Quality monitoring requires a combination of automated metrics, human evaluation, and production monitoring systems that detect quality degradation over time.

What you will learn in this course

This course is structured to take you from foundational concepts to designing complete GenAI systems:

Section 1 (this page): Introduction and overview of GenAI system design.

Section 2: Fundamental Concepts. Deep dives into the ML building blocks you need: transformers, attention, tokenization, evaluation metrics, parallelism strategies, inference optimization, RAG, and finetuning.

Section 3: Back-of-the-Envelope Calculations. How to estimate GPU memory requirements, training time, training cost, inference latency, and serving costs. These calculations are critical for interview credibility.

Section 4: The SCALED Framework. A detailed walkthrough of the 6-step framework with examples and interview strategies.

Sections 5-9: Case Studies. Complete system designs for five different GenAI modalities:

• Text-to-text (ChatGPT-like systems)

• Text-to-image (DALL-E / Stable Diffusion-like systems)

• Text-to-speech (ElevenLabs-like systems)

• Text-to-video (Sora-like systems)

• Image captioning (multimodal understanding systems)

Each case study covers both the training infrastructure and the deployment architecture.

Section 10: Conclusion. Key takeaways, study plan, and practice problems.

Who this course is for

This course is designed for:

• ML engineers who understand models but want to learn how to design full systems around them

• Software engineers transitioning into AI/ML roles who need to understand the ML-specific components

• Engineering managers who need to make informed decisions about GenAI infrastructure

• Interview candidates preparing for GenAI system design interviews at top companies

Prerequisites: Basic understanding of machine learning concepts (what a neural network is, what training means) and basic system design knowledge (load balancers, databases, APIs). We review the key ML concepts in Section 2, but this is not an intro to ML course.

How to get the most out of this course

Here are some practical tips:

Read with a whiteboard. For each case study, try drawing the architecture diagram yourself before reading the solution. This simulates the interview experience.

Practice the calculations. The back-of-the-envelope section is one of the highest-value parts of this course. Being able to quickly estimate that "a 70B parameter model needs about 140GB in FP16, so we need at least 2 A100-80GBs" is the kind of quantitative reasoning that impresses interviewers.

Focus on tradeoffs, not memorization. Interviewers do not expect you to remember exact numbers. They want to see that you understand the tradeoffs: why would you use quantization vs distillation? When is RAG better than finetuning? What are the pros and cons of continuous batching?

Practice explaining out loud. GenAI system design interviews are collaborative conversations. Practice explaining your designs verbally, as if you were talking to a colleague.

Interview tip: The best candidates do not just describe components. They explain WHY they chose each component, what alternatives they considered, and what tradeoffs they made. This is what separates a senior answer from a junior one.

Let us get started. In the next section, we will build your foundation by reviewing the key machine learning concepts that underpin every GenAI system.