Fine-Tuning in 2026: LoRA vs QLoRA and the 4-Step Workflow

Every team asks the same question in 2026: fine-tune, build RAG, or just write better prompts? The honest answer is that most "fine-tuning projects" are really prompt problems wearing a GPU bill. Fine-tuning earns its keep when you need consistent format, tone, or behavior that prompting can't pin down—and when you have a few thousand clean examples to prove it.

First, rule out RAG

Fine-tuning teaches behavior. RAG supplies knowledge. If your model needs to cite this quarter's pricing or internal docs, fine-tuning won't help—it bakes in stale facts. Fine-tune when you want reliable structured output, domain-specific reasoning patterns, or a smaller model to replace a bigger one on a narrow task.

LoRA and QLoRA, without the math

Full fine-tuning updates every weight. LoRA freezes the base model and trains small adapter matrices, so you touch roughly 1% of parameters and ship a file measured in megabytes. QLoRA goes further: it loads the base model in 4-bit precision, then trains LoRA adapters on top. That's how people fine-tune 70B models on a single 48GB card—slower, slightly lossier, dramatically cheaper.

| Method | Memory | Best for | |---|---|---| | Full FT | Very high | Rare; max quality | | LoRA | High | 7B–13B on one A100 | | QLoRA | Low | 70B on consumer/prosumer GPUs |

Start with QLoRA. If quality plateaus and you have the hardware, move up.

The four-step workflow

1. Prepare data. This is 70% of the job. A few thousand high-quality examples beat 100k scraped ones. Deduplicate, strip PII, and hold out a clean eval set before you look at training data. Format consistently—chat templates or instruction/response pairs.

2. Train. Pick a base model that already fits your domain. Set a low learning rate, watch eval loss, and stop early. Overfitting shows up fast on small datasets.

3. Evaluate. Don't trust loss curves alone. Run task-specific evals, A/B against the base model, and have humans review a sample. Check for regressions on general capabilities.

4. Deploy and monitor. Merge adapters or serve them hot-swappable. Log real outputs and watch for drift—your data distribution will shift.

A minimal QLoRA starting point

from peft import LoraConfig
config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                    target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")

Your move this week

Pick one narrow task, collect 500 clean examples, and run a QLoRA experiment against your current prompt-only baseline. Measure both. If fine-tuning doesn't beat prompting on your eval set, you just saved yourself a quarter of GPU spend—and you'll know exactly why.