PEFT Interview Questions: LoRA, Adapters, Quantization, and Fine-Tuning Trade-Offs

PEFT interview questions are rarely satisfied by expanding the acronym. Interviewers want to know whether you can derive the efficiency claim, distinguish methods that are often blurred together, and choose an adaptation strategy under real data, hardware, latency, and deployment constraints.

A strong answer moves from mechanism to numbers to production consequences. This guide gives you that answer structure for LoRA, bottleneck adapters, quantization, QLoRA, and full fine-tuning without turning the interview into a list of library options.

Use PracHub's LLM interview questions to practice explaining the trade-offs aloud. The question-bank records are practice material, not a prediction of the exact questions in your interview.

PEFT interview questions comparing LoRA adapters quantization and full fine-tuning

Quick answer: how should you compare PEFT methods?

Start with what is trainable, then compare memory, quality, latency, and operational complexity. Full fine-tuning updates the pretrained parameters. Classic adapters freeze the base and insert small trainable modules. LoRA freezes the base and learns a low-rank update to selected weight matrices. Quantization changes numerical representation; it is not, by itself, a fine-tuning method. QLoRA combines a frozen quantized base with trainable LoRA weights.

Method	What changes during training?	Main reason to choose it	Main cost or risk
Full fine-tuning	All or most model weights	Maximum adaptation capacity	High training memory and a full checkpoint per variant
Bottleneck adapter	Small inserted neural modules	Modular task-specific components	Extra operations in the forward path
LoRA	Low-rank updates on selected matrices	Small trainable and storable deltas	Rank and target-module choices can limit capacity
QLoRA	LoRA weights over a frozen low-bit base	Lower base-model memory during adaptation	Quantization error and kernel/tooling constraints

Do not claim that one method always wins. The best choice depends on the size and quality of the adaptation data, distance from the base model's domain, available accelerators, number of task variants, and serving model.

Use a five-part interview answer framework

First, clarify the objective. Is this supervised instruction tuning, domain adaptation, preference optimization, personalization, or compression for serving? Ask how much data exists, whether the base model already performs the task, and which regressions are unacceptable.

Second, name exactly what is frozen and trainable. Third, derive the trainable parameter count and identify the remaining memory consumers: frozen weights, gradients, optimizer states, activations, temporary buffers, and distributed-training overhead. PEFT reduces several terms, but it does not make the base model or activations disappear.

Fourth, explain the serving design. Will one adapter be merged into one base, or must a service hot-swap hundreds of tenant adapters? Fifth, propose an experiment with quality, regression, latency, throughput, memory, and cost metrics. This order turns a textbook definition into an engineering answer.

PEFT decision framework routing domain shift training memory task modularity and serving footprint to full fine-tuning LoRA adapters quantization and QLoRA

Derive LoRA instead of only defining it

Suppose a pretrained linear layer has W in R^(d_out x d_in). Full fine-tuning learns every element of W, so that layer contributes d_out * d_in trainable parameters. LoRA keeps W frozen and writes the update as delta_W = (alpha / r) * B * A, where A in R^(r x d_in) and B in R^(d_out x r).

The LoRA update therefore has r * (d_in + d_out) trainable parameters. For a square 4,096-by-4,096 projection with rank 16, full updating needs 16,777,216 parameters, while LoRA needs 131,072. That is about 0.78% for that matrix. It is a local calculation, not the trainable percentage for the entire model; the total also depends on the layers and modules targeted, any trained biases, embeddings, or output head.

The rank is a capacity constraint, not a quality dial that improves forever. A low rank assumes the useful task update lies in a lower-dimensional subspace. Higher rank increases parameters and compute and may help until data, optimization, or the base model becomes the bottleneck. Treat rank, target modules, scaling, learning rate, and data quality as experimental choices.

At inference, a static LoRA update can often be merged into the base weight as W' = W + delta_W, avoiding separate adapter matrix multiplications. Keeping it unmerged supports switching or composing adapters, but then adapter loading, batching, memory locality, and latency become serving concerns.

Explain how bottleneck adapters differ from LoRA

A classic bottleneck adapter inserts a small residual module into the network. A hidden vector of width d is projected down to bottleneck width m, passed through a nonlinearity, projected back to d, and added to the residual stream. Ignoring biases, one such module adds about 2 * d * m trainable parameters.

Both adapters and LoRA freeze the base, produce small task-specific artifacts, and make multi-task storage attractive. Their placement differs. The bottleneck adapter transforms activations through a new nonlinear path, while LoRA changes the effective linear map through a factored additive update. A classic nonlinear adapter is not generally merged into one existing weight matrix the way LoRA can be.

That distinction matters in production. Adapters are explicitly modular and may be useful when task routing and architectural separation matter. LoRA is often attractive when a mergeable delta and a mature tooling path matter. The interview answer should connect the mechanism to the serving requirement, not declare a universal winner.

Separate quantization from QLoRA

Quantization stores or computes values with fewer bits. It can reduce model memory and memory bandwidth, and supported kernels may improve throughput. It can also introduce approximation error, require scales or other metadata, and perform poorly when the hardware or runtime lacks an efficient kernel. Bit width alone is not a quality or latency guarantee.

A useful first estimate is weight storage only. Seven billion parameters require about 14 GB at 16 bits, 7 GB at 8 bits, or 3.5 GB at 4 bits using decimal units. Real training and serving footprints are higher because of quantization metadata, non-quantized tensors, caches, activations, workspaces, fragmentation, and framework overhead.

QLoRA backpropagates through a frozen 4-bit quantized base into trainable LoRA adapters. The base weights are not being updated as crude 4-bit values; they remain frozen, are dequantized to a compute type for operations, and gradients update the adapter parameters. The original QLoRA work introduced NormalFloat 4-bit weights, double quantization of quantization constants, and paged optimizers, and demonstrated 65-billion-parameter fine-tuning on one 48 GB GPU. Present that result as a paper demonstration, not a promise for every model, sequence length, batch size, or software stack.

Choose with a production decision rubric

Constraint	Good starting point	What to verify
One task, ample compute, large high-quality dataset, broad domain shift	Full fine-tuning baseline	Whether added capacity beats PEFT enough to justify cost and regressions
Many tasks or tenants sharing one base	LoRA or adapters	Artifact isolation, routing, loading time, batching, and version compatibility
Training is limited mainly by base-weight memory	QLoRA	Quality against higher precision, supported kernels, and end-to-end peak memory
Strict latency with one stable adapter	Merged LoRA or a consolidated checkpoint	Merge precision, deployment format, throughput, and rollback path
Frequent hot-swapping or personalization	Unmerged LoRA or modular adapters	Cache policy, cold-load latency, per-request routing, and mixed-adapter batching

Full fine-tuning has more freedom to change the model and may be justified for severe domain shift or continued training at scale. It also creates gradients and optimizer state for far more parameters and usually requires a full checkpoint per variant. PEFT is strongest when the base already contains useful capabilities and the desired change is narrower.

For serving, calculate the whole system. A tiny adapter does not help if every worker duplicates a large base unnecessarily, while a shared base with routed adapters can greatly reduce per-task storage. Conversely, dynamic adapter execution can complicate batching. Benchmark the exact runtime, precision, sequence lengths, concurrency, and hardware that will ship.

Design an evaluation that can change the decision

Compare methods on identical train, validation, and regression sets, with budgets that are fair and clearly stated. Report the primary task metric and slices for rare cases, long contexts, important languages or domains, safety behavior, calibration when probabilities drive decisions, and retention of general capabilities. Include multiple seeds when variance could change the conclusion.

Measure system outcomes too: peak training memory, wall-clock time, accelerator-hours, checkpoint size, load time, p50 and p95 latency, throughput, and cost at expected concurrency. If LoRA is merged, compare the merged artifact. If adapters are routed dynamically, exercise realistic cache misses and mixed traffic.

Finally, define promotion and rollback. Version the base model, tokenizer, adapter configuration, data, code, and evaluation report together. A canary or shadow test should detect production regressions before broad rollout. The interview signal is not that you know every benchmark; it is that your experiment can disprove your preferred option.

Practice PEFT interview questions on PracHub

These PracHub question-bank records train the explanation and system-design skills behind this topic. They are not predictions of your exact assessment or interview.

PracHub question	Practice focus	Why it helps
Compare Losses and Explain LoRA	LoRA mechanism and reasoning	Tests whether you can move from definition to a precise technical explanation.
Explain LLM architecture, tuning, evaluation	End-to-end model adaptation	Connects architecture choices to tuning and evidence.
Design an AWS fine-tuning platform for LLMs	Training-platform design	Turns PEFT constraints into storage, scheduling, deployment, and observability decisions.
Explain Model Compression Techniques	Quantization and compression	Builds a careful comparison of memory, quality, and hardware trade-offs.
Explain LLM fundamentals and trade-offs	Trade-off communication	Practices choosing an approach from constraints instead of reciting names.

Avoid common PEFT interview mistakes

Calling quantization a trainable adapter. Quantization changes representation; QLoRA adds a PEFT training mechanism on top of a quantized base.
Counting only trainable parameters. Peak memory also includes the frozen base, activations, temporary buffers, and runtime overhead.
Quoting one memory multiplier as universal. Precision, optimizer, sharding, checkpointing, sequence length, batch size, and implementation change the result.
Saying LoRA has zero latency in every deployment. A merged static adapter can avoid extra operations; an unmerged multi-adapter service has different behavior.
Assuming PEFT matches full fine-tuning everywhere. Original papers show strong results on their evaluated settings, not a guarantee for every dataset and distribution shift.

PEFT interview questions: FAQ

Is LoRA the same as an adapter?

LoRA is an adapter-style PEFT method, but it is not the same mechanism as the classic bottleneck adapter. LoRA learns a factored additive weight update; a bottleneck adapter inserts a small nonlinear activation path.

Does a higher LoRA rank always improve quality?

No. Higher rank increases the update's capacity and parameter count, but benefits can plateau or reverse depending on data, regularization, target modules, and optimization. Select rank with held-out evidence.

Is QLoRA full 4-bit training?

No. In QLoRA, the pretrained base is frozen and stored in 4-bit form, while LoRA parameters are trained and computation uses a suitable higher-precision compute type. That distinction is central to a correct answer.

Can every LoRA adapter be merged for inference?

Standard LoRA updates can often be folded into compatible base weights, but support depends on the model, quantization format, adapter variant, and serving stack. Merging also gives up easy hot-swapping unless separate artifacts are retained.

When should you prefer full fine-tuning?

Consider it when you have enough high-quality data and compute, need broad model change, and observe a meaningful quality advantage over well-tuned PEFT baselines. The decision still needs regression tests and a serving-cost calculation.

Final takeaway

The best PEFT interview answer is a compact design review: define the adaptation goal, state what is trainable, derive the parameter and storage implications, connect them to serving, and propose an evaluation that could overturn your choice. Practice that sequence until you can defend LoRA, adapters, QLoRA, or full fine-tuning from the same set of constraints.

Then test the framework on PracHub's LLM interview practice: give your answer first, calculate one concrete example, and only then compare it with the solution.

Sources and Further Reading

Research note: This guide was checked on August 24, 2026. Library behavior and hardware support can change, so verify the current documentation for the stack you will use.