LoRA: Why It Is Needed and How to Count Adapter Parameters at Rank 8

Answer three questions on LoRA: what it is and why it is needed, how many parameters a LoRA adapter has for a 500 × 1000 weight matrix at rank 8, and how you would estimate the size of a LoRA adapter for a model of about one billion parameters at rank 8. ### Clarifying Questions - Is the goal one fine-tuned model, or many task-specific variants of the same base model? - Should sizes be given in parameters, in bytes, or both? ### Part 1 — What LoRA is and why it is needed Explain what LoRA is and why it is needed. ```hint Freeze and add Think about what is trained and what must be stored for each task when the pretrained weights never change. ``` #### What This Part Should Cover - The low-rank update, its initialization and scaling, and what stays frozen - Where the savings come from: optimizer memory, storage per task, inference cost - Limitations and common variants ### Part 2 — One 500 × 1000 matrix at rank 8 Calculate the number of parameters of a LoRA adapter for a 500 × 1000 weight matrix with rank 8. ```hint Two factors Write the shapes of the two low-rank factors whose product must have the shape of the original matrix. ``` #### Clarifying Questions for this Part - Is the matrix stored as (output, input) or (input, output), and does that change the count? #### What This Part Should Cover - The shapes of the two factors and the parameter count - A comparison with the full matrix - Whether anything else, such as a bias or the scaling factor, adds trainable parameters ### Part 3 — An adapter for a model of about one billion parameters How would you estimate the size of a LoRA adapter at rank 8 for a model with roughly one billion parameters? ```hint Per matrix, then per layer Work out the adapter cost of one adapted matrix from its two dimensions, then count the adapted matrices in a layer and the layers in the model. ``` #### Clarifying Questions for this Part - Which weight matrices receive adapters: only some attention projections, all of them, or the MLP layers too? - Which architecture (number of layers, hidden size) and which storage precision should be assumed? #### What This Part Should Cover - An explicit assumed architecture and set of adapted matrices - The count per matrix, per layer and in total, in parameters and in bytes - A sanity check of the result as a fraction of the base model ### What a Strong Answer Covers - The LoRA update, its initialization and scaling, and what is frozen - The memory and storage savings, and where they come from - A correct parameter count for the 500 × 1000 example, with the factor shapes - A transparent, assumption-driven estimate for the one-billion-parameter model, with a sanity check - Trade-offs: rank, which modules to adapt, merging, and variants such as QLoRA ### Follow-up Questions - How do the rank and the choice of target modules affect quality and adapter size? - Can the adapter be merged into the base weights, and when would you keep it separate? - What does QLoRA change, and what does it save? - Why is one of the two factors initialized to zero?