To fine-tune an open-weight model, choose a base model, prepare a few hundred to a few thousand high-quality examples of the input and output you want, train small adapter layers with LoRA or QLoRA instead of the whole model, evaluate the result against the base model on a held-out test set, then merge or serve the adapter with a tool like vLLM. This guide explains each step and when fine-tuning is worth it.
When should you fine-tune, and when not?
| Goal | Best approach |
|---|---|
| Answer from your documents or data that changes | RAG |
| Follow a consistent format, style or tone | Fine-tuning |
| Do one narrow task cheaply at high volume (classification, extraction) | Fine-tune a small model |
| Use tools or a domain vocabulary more reliably | Fine-tuning, often combined with RAG |
| Quick improvement with little data | Better prompts and examples first |
Fine-tuning teaches a model how to respond; RAG gives it what to know. Many production systems use both.
LoRA and QLoRA explained
- Full fine-tuning updates every weight. It needs a lot of GPU memory and produces a full copy of the model.
- LoRA (Low-Rank Adaptation) freezes the original weights and trains small extra matrices. It needs far less memory, trains faster and produces a small adapter file.
- QLoRA loads the base model in 4-bit precision and trains LoRA adapters on top, so mid-size models can be fine-tuned on a single GPU.
Step-by-step fine-tuning process
1. Define the task and success metric
Write down exactly what the model should do and how you'll judge it, for example "extract these 12 fields as valid JSON with at least 95% field accuracy on our test set".
2. Choose a base model
Start with the smallest model that could plausibly do the job, usually an instruction-tuned version. Qwen 3 offers many sizes, and Llama, DeepSeek and GLM families are common choices. Compare the options in our open-weight models guide, and check each licence.
3. Prepare the dataset
- Collect real examples: inputs paired with ideal outputs, in the model's chat format.
- Quality beats quantity. A few hundred clean, consistent examples often outperform thousands of noisy ones.
- Remove personal data you don't need, duplicates and contradictory examples.
- Hold back 10 to 20% as a test set the model never sees during training.
4. Train
| Tool | Why use it |
|---|---|
| Hugging Face TRL + PEFT | The standard libraries for supervised fine-tuning (SFT), preference tuning (DPO) and LoRA |
| Unsloth | Faster, memory-efficient LoRA and QLoRA training on a single GPU |
| Axolotl | Configuration-file driven training for repeatable runs |
| Managed platforms | Cloud fine-tuning services when you don't want to manage GPUs |
Start with modest settings (a few epochs, a small learning rate) and watch the validation loss to avoid overfitting.
5. Evaluate
- Compare the fine-tuned model with the base model and a prompted larger model on your test set.
- Check for regressions: general instructions, safety behaviour and formats it handled before.
- Have a domain expert review a sample of outputs.
6. Deploy
- Serve the base model with the LoRA adapter, or merge them into one model.
- Use vLLM or SGLang for production throughput, or quantise to GGUF for llama.cpp and Ollama.
- Monitor quality in production and retrain periodically with new, corrected examples.
Common fine-tuning mistakes
- Fine-tuning to add knowledge that should come from RAG.
- Training on inconsistent examples, which teaches inconsistent behaviour.
- No held-out test set, so improvements can't be proven.
- Ignoring the base model's licence and data-protection rules for training data.
Key takeaways
- Fine-tune for behaviour, format and narrow tasks; use RAG for knowledge.
- LoRA and QLoRA make fine-tuning affordable on a single GPU.
- Data quality and a fair evaluation decide whether fine-tuning was worth it.
AppifyDevs fine-tunes and deploys open-weight models for private, cost-efficient AI. Tell us about your use case.