violet = frozen · glowing = trainable · pink panels = LoRA A/B · teal orbs = soft tokens
Training knobs (they update the math above)
LoRA replaces the weight update ΔW (d×d) with two skinny matrices B·A of shapes d×r and r×d. Trainable params scale linearly with r — rank 8 on a 7B model touches ≈0.1% of weights.
Full FT wants tiny LRs (1e-5 range) or it bulldozes pretrained knowledge; LoRA tolerates ~10× higher because the base is frozen.
Rule of thumb: hundreds of high-quality examples already move behavior for narrow tasks; full-domain adaptation wants 10k–1M+. Quality beats volume.
Interview drill
Three questions in the style of the original thread. Click an answer — wrong picks explain why.
Q1. What is fine-tuning in LLMs?
Fine-tuning starts from pretrained weights and continues gradient descent on new, narrower data. From-scratch training is pretraining; weight removal is pruning.
Q2. Why is LoRA called “parameter-efficient”?
LoRA freezes W and learns ΔW = B·A with rank r ≪ d. At inference you can merge W + B·A, adding zero latency. (4-bit quantization + LoRA = QLoRA, a different but related trick.)
Q3. What is catastrophic forgetting and which method risks it most?
When every weight is free to move, narrow-domain gradients overwrite broadly useful representations. Mitigations: lower LR, mixing in general data (replay), or freezing most weights — which is exactly what LoRA and prompt-tuning do.
Score: 0 / 3