Fine-tuning Open-Weight Models: LoRA, QLoRA, and Production Export

Why fine-tune open-weight models?

Frontier models (Claude, GPT-4, Gemini) are expensive and closed. Open-weight models (Llama 3.1, Qwen 2.5, DeepSeek) are free, auditable, and can run locally. But they are often trained on generic data and underperform on specialized tasks: customer support tone, technical documentation writing, domain-specific code generation, or niche classification tasks.

Fine-tuning takes a pre-trained open-weight model and updates its weights using your task-specific data. Done well, a fine-tuned 7B model can outperform a generic 70B model on your specific use case — at a fraction of the compute cost and with zero data leaving your infrastructure.

When NOT to fine-tune: If your data is tiny (under 100 examples), prompt engineering or in-context learning often works better. If you need reasoning on novel problems, frontier models still win. If your data is proprietary and model IP matters, fine-tuning is a way to keep training in-house.

Core architectures: LoRA vs QLoRA vs full fine-tuning

Full fine-tuning updates every parameter in the model. It is memory-intensive and slow. LoRA (Low-Rank Adaptation) trains only small adapter matrices, dramatically cutting memory use. QLoRA quantizes the base model first, then applies LoRA, enabling fine-tuning on consumer hardware.

Practical recommendation: Start with QLoRA on an 8GB NVIDIA GPU (RTX 3060, RTX 4060, or similar). If you have 24GB+ VRAM, LoRA is slightly faster. Full fine-tuning is rarely necessary for open-weight models unless you are doing research or have unlimited compute.

Memory optimization techniques

Gradient checkpointing

Gradient checkpointing recomputes intermediate activations during backpropagation instead of storing them. It trades ~20-30% training speed for ~30-40% memory savings. Essential when VRAM is tight.

gradient_checkpointing_enable() in Unsloth or set gradient_checkpointing: true in Axolotl config.

Flash Attention 2

Flash Attention is an IO-aware attention algorithm that speeds up training and cuts memory use by ~20%. Most modern libraries (Unsloth, transformers) auto-enable it. Enable explicitly in Axolotl with use_flash_attention: true.

4-bit quantization (QLoRA)

Load the base model in 4-bit using bitsandbytes. Roughly 75% memory savings vs. loading in float16. Apply LoRA adapters on top. When merged post-training, the fine-tuned model retains full precision weights or can be re-quantized.

Batch size and gradient accumulation

Smaller batch size = lower VRAM but noisier training. Gradient accumulation simulates a larger batch by accumulating gradients over multiple steps before updating. Example: batch_size=2, gradient_accumulation_steps=4 simulates batch=8.

Tooling stack: Unsloth vs Axolotl

Two dominant frameworks for fine-tuning open-weight models on consumer hardware: Unsloth for speed and simplicity, Axolotl for production pipelines.

Unsloth quickstart

Install unsloth, load a model with FastLanguageModel.from_pretrained(), define a training loop using HuggingFace Trainer, and export with model.save_pretrained(). Typical 7B fine-tune on 8GB GPU takes 1-3 hours depending on data size.

Axolotl quickstart

Write a YAML config file specifying model, dataset, LoRA rank, batch size, and checkpointing strategy. Run axolotl train config.yaml. Axolotl handles data loading, logging (WandB), multi-GPU orchestration, and checkpoint management. Steeper setup curve but scales to large teams and datasets.

Compute environment setup

Local hardware (single GPU)

An NVIDIA GPU with 8GB+ VRAM (RTX 3060, RTX 4060, A2000, L4) is sufficient for QLoRA fine-tuning. Install CUDA, cuDNN, and PyTorch with CUDA support. A multi-GPU setup (2x RTX 4090) speeds training but requires distributed training setup (Axolotl handles this).

Cloud GPU instances (RunPod, Lambda Labs, Vast.ai)

For larger models or frequent fine-tuning, rent GPU instances on-demand. RunPod offers competitive hourly pricing for NVIDIA GPUs (RTX 4090, A100) and pre-configured PyTorch pods. Upload your data, spin up an instance, fine-tune in 1-2 hours, download the adapter weights, and power down. Cost: $0.20–$1.50/hour depending on GPU tier.

Explore RunPod for on-demand GPU infrastructure →

Environment setup (local or cloud)

1. Create a venv and install dependencies:pip install unsloth[colab-new] xformers bitsandbytes
2. Prepare your dataset: CSV or JSON with "text" or "instruction"/"input"/"output" fields. Minimal: 100+ examples of your domain-specific task.
3. Load model and configure LoRA rank: Typical rank=16, lora_alpha=32. Higher rank = more learnable parameters but more memory.
4. Train: 1–5 epochs depending on data size. Monitor loss and validation accuracy.

Data preparation and training

Data format

Unsloth accepts any HuggingFace-compatible dataset. Axolotl prefers instruction/input/output format (alpaca-style). Example row:

Training hyperparameters

• Learning rate: 1e-4 to 5e-4. Lower for domain-specific fine-tuning. Start conservative; tune down if loss diverges.
• Batch size: 2-4 on 8GB GPU, 8-16 on 24GB. Smaller batch = more gradient noise but more VRAM-efficient.
• Epochs: 1-3 depending on dataset size. One epoch on 1K examples = ~1 hour. Validation loss plateaus quickly; avoid overfitting.
• Warmup steps: 10% of total steps. Prevents gradient shock early in training.

Validation and evaluation

Hold out 10% of your data for validation. Monitor validation loss and perplexity. After training, sample predictions and compare against a baseline model. Use task-specific metrics (BLEU, F1, ROUGE) if applicable.

Export: Merging adapters and quantizing for inference

Step 1: Merge LoRA adapters into the base model

After training, you have two separate checkpoints: frozen base weights (e.g., Llama 3.1 7B) and trained LoRA adapters (~50 MB). Merging combines them into a single model file.

The merged model is now a standard HuggingFace model checkpoint ready for quantization or direct serving.

Step 2: Quantize to GGUF or EXL2

Quantization compresses model weights to lower precision (e.g., 4-bit) for inference. Two formats:

For local serving (Ollama, llama.cpp): Convert to GGUF using llama.cpp/convert.py merged_model/ --outfile model.gguf. Then quantize: quantize model.gguf model-q4.gguf Q4_K_M.

For vLLM serving (cloud-friendly): Use ExLlama2 quantization. Slightly faster inference with similar quality at Q4.

Step 3: Serve locally with Ollama or vLLM

Place the quantized model file (GGUF or EXL2) in Ollama's model directory, create a Modelfile, and run ollama run my-model. Or use vLLM with python -m vllm.entrypoints.openai_api_server --model my-model for OpenAI-compatible inference.

Common pitfalls and best practices

• Overfitting on small datasets: 1 epoch is often enough. Use early stopping. Validate on held-out data.
• Loss diverges: Learning rate too high. Reduce by 2–5x. Ensure warmup steps are set.
• Merged model performs worse than base: Low-quality training data or insufficient epochs. Curate data carefully; balance quality over quantity.
• CUDA out of memory (OOM): Reduce batch size, enable gradient checkpointing, or use 4-bit quantization (QLoRA).
• Quantized model quality drops too much: Use Q5 or Q6 instead of Q4. Test on your specific task.
• Inference is slow: Ensure VRAM-resident inference (model fits entirely in GPU memory). Check batch size during serving.

End-to-end example: Fine-tuning Llama 3.1 8B for customer support

You have 500 customer support exchanges and want a fine-tuned model that responds in your company's tone.

1. Prep data: Format as instruction/input/output JSON. 450 train, 50 validation.
2. Set up Unsloth: pip install, load Llama 3.1 8B with 4-bit quantization.
3. Configure LoRA: rank=16, lora_alpha=32, lora_dropout=0.05.
4. Train: 3 epochs, batch=2, lr=2e-4, gradient_accumulation=2. Runs in ~4 hours on RTX 4090 or ~8 hours on RTX 3060.
5. Merge and quantize: Merge LoRA adapters, convert to GGUF, quantize to Q4_K_M.
6. Serve: Load in Ollama, hit it with your test support tickets. Evaluate on response quality and tone.
7. Iterate: If quality is low, curate data further or increase epochs. If overfitting, reduce epochs or add validation regularization.

Summary

Fine-tuning open-weight models is now accessible to individual developers and small teams. QLoRA on an 8GB GPU opens the door to domain-specific model adaptation without frontier model costs or data leaving your infrastructure. Unsloth enables fast, simple fine-tuning. Exporting to GGUF or EXL2 makes inference efficient locally or at scale.

Start with small-scale experiments: 100–200 examples, 3 epochs, QLoRA on a consumer GPU. Validate on a held-out test set. Iterate on data quality. Once your fine-tuned model outperforms the base model on your task, it is ready for production serving.

FAQ