Key concepts in designing generative AI systems

Follow the representations used between a prompt and a generated result: tokens, embeddings, image features, phonemes, acoustic features, and scene relationships. Learn what each representation makes explicit and what the next component still has to infer.

A generation pipeline changes the representation of information several times. A tokenizer produces text units; an embedding layer supplies vectors; a speech system may predict acoustic features before a vocoder produces a waveform. These are different interfaces, not interchangeable ways to “understand” the input. This lesson introduces the interfaces used in the later design cases.

Representations in this lesson

For each interface, ask what enters, what leaves, and which information could be lost or inferred incorrectly.

Tokenization

Tokenization splits text into units that a model maps to token IDs. The units depend on the tokenizer: words, subword pieces, characters, or bytes are common choices. A hypothetical word-level split of “The five boxing wizards jumped quickly” yields six word tokens. It does not predict how a particular LLM’s tokenizer will split that sentence.

Tokenization techniques

The main choices trade vocabulary size against sequence length and the handling of unfamiliar inputs:

• Word-level: Keep complete words as units. This can require a large vocabulary and needs a policy for unfamiliar words and languages without explicit word separators.

• Subword: Use reusable pieces that need not correspond to grammatical prefixes or stems. “Unbelievable” might split into several pieces, but the exact split depends on the learned vocabulary. BPE is one method; SentencePiece supplies language-independent subword tokenization and can train directly from raw sentences.

• Character-level: Use character units, often reducing vocabulary size at the cost of longer sequences. Byte-based schemes make a related but distinct choice.

• Speech inputs: Some TTS pipelines map text to phoneme tokens; others accept characters or other units. Pronunciation conversion is a separate linguistic task, not a requirement of every tokenizer.

• Multimodal inputs: Images and audio may become patch features, continuous representations, or discrete codes depending on the model. CLIP models align separate text and image encoders in a shared feature space; they do not establish one common tokenizer for both modalities.

Embedding

An embedding maps an input to a numerical vector that a model can process. A token’s initial embedding is different from its context-dependent representation after transformer layers, and both differ from a sentence vector used for retrieval. Specify which representation a component consumes before choosing how to store or compare it.

Embedding techniques

Some well-known embedding techniques are:

• Word2Vec: Learns word vectors through context-prediction objectives, including continuous bag of words and skip-gram.

• GloVe: Learns word vectors using corpus-wide word co-occurrence statistics.

• FastText: Incorporates subword information so representations can use pieces of unfamiliar words.

• Universal Sentence Encoder: Produces sentence-level vectors. Its original paper presents two encoder variants with different resource and accuracy trade-offs, so the name does not imply a single transformer architecture.

• InferSent: Learns sentence representations for transfer to sentence-level tasks.

• BERT representations: Produce context-dependent token features. A sentence-retrieval system also needs a suitable way to construct and evaluate a sentence representation.

Nearby vectors indicate similarity according to the representation and distance measure, not a guarantee of interchangeable meaning. “Cool” may concern temperature or style; the relevant sense depends on context. A retrieval service may index document vectors in a vector store, while a language model’s token embedding table is normally part of the model itself.

Embeddings represent information for computation. The next interface instead changes the resolution of the visual output.

Image resolution enhancement techniques

Increasing resolution increases the number of pixels. It does not guarantee that new details match the original scene or the prompt. A generative upscaler predicts plausible missing detail, which can look convincing while being incorrect.

Three approaches illustrate the design choices:

• Super-resolution GANs: A generator predicts a higher-resolution image while a discriminator helps train it toward realistic-looking outputs. SRGAN and later ESRGAN illustrate this approach. Predicted texture is not verified recovery of information absent from the input.

• Perceptual losses: Compare features from a pretrained network, such as VGG, instead of relying only on pixel differences. The chosen features influence which errors training penalizes. Perceptual similarity and exact reconstruction are different objectives.

• Transformer-based restoration: Models such as SwinIR use attention-based feature processing for reconstruction. This does not require treating upscaling as autoregressive text-like sequence prediction, nor guarantee smoother results for every input.

Speech pipelines introduce another representation choice: whether to predict pronunciation units before acoustic features.

Text-to-phoneme conversion

Phonemes are sound categories that distinguish words within a language. A text-to-phoneme component maps written input to a pronunciation representation. In “The wind will wind the clock,” the noun uses /wɪnd/ and the verb uses /waɪnd/. The same spelling has two readings, so the conversion needs context. These are homographs whose pronunciations differ.

A pronunciation system may combine dictionaries, linguistic rules, and learned prediction. It must handle ambiguity and unfamiliar names rather than assume that a model family guarantees correct pronunciation.

• Sequence-to-sequence models: Map a text sequence to a pronunciation sequence, accommodating different input and output lengths.

• Attention: Helps an encoder-decoder model select relevant input information during prediction, but the training task still must teach the intended mapping.

• Contextual language models: BERT-, GPT-, or T5-style components can be adapted to support contextual decisions. Their names alone do not define a working phoneme converter or its supported languages.

Acoustic model in text-to-speech systems

In a common two-stage TTS pipeline, an acoustic model predicts features such as a mel-spectrogram from text or linguistic input, then a vocoder converts those features to a waveform. Duration, pitch, and energy affect the resulting speech. The model learns these relationships from paired text and speech under its chosen representation and objective.

Tacotron 2 predicts mel-spectrograms from character embeddings and uses a WaveNet vocoder. FastSpeech 2 explicitly incorporates duration, pitch, and energy information. Other families, including Fish Speech, use their own components and intermediate representations. Inspect the selected architecture rather than assuming every TTS model implements the same acoustic-model/vocoder boundary.

Visual-generation pipelines can also expose an explicit plan before producing pixels.

Concept extraction from prompts

Concept extraction makes selected prompt requirements explicit. A planner might identify entities, attributes, actions, and spatial relationships before generation. This is an optional design choice: a CLIP text embedding is a learned vector, not automatically a structured list of those concepts. Consider the hypothetical prompt “A Siberian husky dog sitting on a red chair in a cool living room.”

A structured plan should retain the supported requirements and preserve unresolved ambiguity. It should not silently turn “cool” into a blue room or low temperature without a stated interpretation. Later generation checks can compare the image with this plan and with the original prompt.

Scene graph generation

A scene graph represents entities as nodes, relationships as edges, and properties as attributes. For the husky example, the graph can distinguish the dog’s breed, the chair’s color, and the “on” relationship. Listing all three words without their relationships would lose that structure.

How is a scene graph generated?

For a text-conditioned design, a parser or planner can turn the extracted requirements into nodes, edges, and attributes. Other scene-graph tasks infer these relationships from images. Here we use a graph built from the prompt as an intermediate specification for generation.

Create Dog, Chair, and Living room nodes. Attach the breed to Dog and the color red to Chair, then connect Dog to Chair with “sitting on” and place the entities in the room. This representation does not need to be a hierarchy: different relationships can connect the same objects. The following figure illustrates a simpler dog-and-chair graph.

The graph makes one potential error easy to state: if red is attached to Dog instead of Chair, the plan has already misbound an attribute before image generation. A correct graph still cannot guarantee a correct image; the generator must use it, and the result must be checked.

As a tracing exercise, reverse the “sitting on” edge or move the red attribute to the dog. Explain how each modified graph changes the requested scene. Then identify which check would catch the mismatch against the original prompt.

Conclusion

Each representation exposes a different part of the task. Tokens define the input units; embeddings provide learned features; phonemes specify pronunciation; acoustic features support waveform generation; and scene graphs can make object relationships explicit. Resolution enhancement changes the visual output while potentially inventing detail.

For a proposed pipeline, label these interfaces and name one error at each boundary. This reveals what the system needs to validate without assuming that transforming data into vectors is equivalent to understanding or satisfying the request.