Inference Optimization in GenAI Systems
In this lesson8 sections
Inference optimization in generative AI systems
Compare inference optimizations by the work they remove: lower-precision arithmetic, fewer model operations, a smaller student, reused computation, or better request scheduling. Measure the quality and latency trade-offs on the workload you intend to serve.
Inference runs a trained model on an input to produce predictions or generated output. Evaluating the model’s outputs tells us about task quality. Measuring the serving path tells us how long users wait and what resources the request consumes.
Once a model meets the initial quality requirement, test it under expected traffic. A model that responds quickly to one request may queue under load or exhaust memory with longer inputs. The service must meet quality, latency, availability, and cost requirements together.
What inference optimization changes
Inference optimization changes the model representation, execution, or serving workflow to reduce resource use or improve performance. Some changes preserve model outputs closely; others deliberately trade some quality for lower cost. Define an acceptable quality threshold and latency budget before selecting a technique.
The right technique depends on the bottleneck. Reducing weight memory helps a memory-limited model, while batching may help underused hardware. Neither fixes a slow external data source by itself.
Inference optimization methods
Begin with techniques that change the model representation, then consider computation reuse and scheduling.
Quantization
Quantization represents selected model values with fewer bits or a more restricted set of numerical levels. Rounding 2.311 to 2.3 is a simple analogy for losing precision, but an actual quantization scheme also defines its scale, range, and representation.
Weights, activations, or cached attention values can be quantized, depending on the method and runtime. Moving from 32-bit floating-point weights to 8-bit values reduces the raw weight payload to one quarter, before accounting for scales and other metadata. The total runtime-memory saving depends on what remains at higher precision.
Lower-precision storage and supported kernels can reduce memory traffic and execution cost. A smaller representation alone does not guarantee faster inference: conversion overhead and missing hardware support can erase the expected gain. Measure the target device rather than inferring latency from file size.
Quality check: Quantization can change outputs. Test representative tasks and difficult cases at each chosen precision; smaller models do not have a general guarantee of negligible accuracy loss.
Pruning
Pruning removes or masks selected weights, neurons, channels, or other model structures. The aim is to reduce work while preserving enough task quality. Removing a whole supported structure can change execution differently from scattering zero weights throughout a dense matrix.
Pruning can degrade task performance, so it may be followed by recovery training or another adaptation step. Its practical value depends on both retained quality and whether the runtime can exploit the resulting structure.
Pruning and quantization are complementary choices, not a universal ranking. PyTorch’s sparsity discussion shows why irregular sparsity can be difficult to accelerate and why hardware-supported patterns matter. Compare measured quality, memory, and latency for the particular combination.
Knowledge distillation (KD)
Knowledge distillation (KD) trains a student model using signals from a teacher, often a larger model or an ensemble. The student learns to approximate useful teacher behavior for the training tasks. The original distillation paper describes this transfer; it is a training procedure, not simply copying a model into a smaller file.
A student can reduce serving cost when it learns enough of the behavior the application needs. It may still lose capabilities, especially outside its training distribution. Distillation therefore does not guarantee that an arbitrarily large GPT-style model can be installed on a phone or that the student retains every teacher capability.
Compare the student with both the teacher and an independently trained small-model baseline. Measure the tasks, latency, peak memory, and energy use that determine whether it is useful on the target device.
Teacher signal: A student can learn from an ensemble of teachers. The resulting serving cost depends on the student, while generating its training signals may require running all teachers.
To read the diagram, separate three signals: the input example, its available ground-truth target, and the teacher’s prediction. The student’s training objective can use the target, the teacher signal, or both.
Trace which model’s weights change after a training step. In ordinary distillation the teacher supplies a fixed signal while the student updates. Then consider a teacher mistake: agreement with the teacher would reproduce that mistake, so held-out task evaluation remains necessary.
Practice: Describe a hypothetical student for an offline writing aid. Specify the teacher outputs it would learn from and one held-out task that could reveal a capability lost during distillation.
Caching strategies
Caching reuses a result or intermediate computation when the new request satisfies the conditions that made the cached value valid. In a generation service, that can mean a completed answer, a shared prompt prefix, or attention keys and values. These caches avoid different work and have different validity rules.
Start by naming the cached object and its key. A similar prompt is not enough if the user, permissions, source data, model version, or generation settings require a different result.
Semantic response cache: Retrieve a prior answer using meaning-based similarity. “What is 1 + 1?” and “What is one plus one?” may be suitable for reuse, but similarity only proposes a candidate. A validity check must decide whether the answer still applies.
Prompt-prefix cache: Reuse intermediate computation for an identical supported prefix. It can reduce prompt-processing work while generating a new answer for the remaining input. This differs from returning a stored response. The Transformers cache guide illustrates reuse of prefilled attention state.
Key-value (KV) cache: Store attention keys and values from earlier tokens during autoregressive generation. Later steps reuse them instead of recomputing those projections. The cache consumes memory that grows with the retained sequence and active requests.
Exact response or result cache: Use a key that identifies the complete relevant request and data context. Redis and Memcached are possible stores. Eviction policies such as LRU or FIFO manage capacity; invalidation rules determine when a result is no longer usable. A cached web result also needs a freshness policy.
Knowledge check
Check your understanding
1 question · source answers hidden
Compare the cached objects and their validity checks:
| Cache | Reused object | Work avoided | Validity question |
|---|---|---|---|
| Semantic response | A previous answer selected by similarity | A new answer-generation path | Does this answer satisfy the new request under the same authority and data version? |
| Prompt prefix | Computation for a supported identical prefix | Repeated prompt processing | Do tokens, model, and runtime conditions permit prefix reuse? |
| KV | Earlier attention keys and values | Recomputing past token projections | Does the state belong to this sequence and model configuration? |
| Exact result | Answer or tool result under a complete key | Repeating the keyed operation | Are the key, permissions, source version, and freshness rules still valid? |
Trace a cache decision: In a hypothetical support service, “Can I return this order?” and “Can I return my other order?” may be semantically similar but refer to different records. A cache hit that ignores the order identity can be fast and wrong. Test false hits as well as hit rate.
Batching
Batching groups compatible requests or inputs for execution. Larger batches can improve throughput by using the accelerator more effectively, but they consume memory and may require requests to wait for companions. Choose batch size and queue-delay limits against the observed traffic pattern.
Latency trade-off: Batch waiting time belongs to each waiting request; it is not divided among the batch. Higher throughput can reduce a backlog under load, but increasing the batching delay can worsen individual or tail latency. Triton’s batching guide recommends measuring throughput within a latency budget.
Use this table to connect each optimization with the measurement that determines whether it helped:
| Technique | Main change | Potential benefit | What to verify |
|---|---|---|---|
| Quantization | Lower-precision representation | Lower memory traffic and storage | Quality, kernel support, total runtime memory |
| Pruning | Remove or mask model structures | Fewer supported operations or values | Retained quality and actual sparse execution |
| Distillation | Train a student using teacher signals | Smaller serving model | Task coverage and behavior outside training examples |
| Caching | Reuse a valid result or computation | Avoid repeated work | False hits, freshness, authority, and memory cost |
| Batching | Schedule compatible work together | Higher hardware utilization and throughput | Queue delay, tail latency, and peak memory |
Conclusion
Use a measured bottleneck to choose an experiment. If weights dominate memory, test quantization. If the same prefix is processed repeatedly, test prefix reuse. If the accelerator is underused while requests queue, test batching. Keep the workload and quality checks fixed so the comparison remains meaningful.
Compression, model partitioning, and hardware-specific kernels provide further options. For each change, record what work it removes, what new cost it introduces, and whether the service still meets its quality and latency requirements. An optimization is useful when that complete comparison improves, not merely when one benchmark number does.