Fine-Tuning and LoRA Workflows: Step-by-Step Processes That Save Time

Fine-tuning teaches style, format and narrow behaviour well. It is a poor way to teach a model new facts that change often. If you need answers based on your current price list, retrieval (feeding relevant documents into the prompt) is cheaper and easier to update.

Good fine-tuning tasks share a pattern: the output format is consistent, you have hundreds or thousands of good examples, and a general model keeps getting the tone or structure slightly wrong even with a careful prompt.

• Good fit: product descriptions in a strict house style, classification into your own categories, extracting fields from messy documents.
• Weak fit: answering questions about frequently updated policies, general research, tasks with fewer than a hundred examples.
• Check first: a well-written prompt with five examples on a strong hosted model. If that reaches your quality bar, stop.

Data quality decides the result more than any hyperparameter. Collect real examples of inputs and the outputs you want, then clean them by hand. Remove contradictions, outdated examples and anything containing personal data you have no right to train on.

Format each example as a prompt and response pair in the chat template your base model expects. Template mismatches are a classic cause of strange outputs after training, so check the model card for the exact format.

Splitting the set

Hold back 10 to 15 percent of examples as an evaluation set that the model never sees during training. Keep it fixed across experiments, otherwise you cannot compare runs. If examples cluster by client or product, split by group so similar items do not leak across the boundary.

Teams that save time write the process down once and reuse it. Common tooling includes Hugging Face Transformers with the PEFT library, TRL for supervised fine-tuning, and wrappers such as Axolotl or Unsloth that reduce setup work. Hosted options exist too; several model providers offer managed fine-tuning where you upload data and receive a tuned model.

• Pick a base model whose licence allows your use and whose size fits your hardware and latency budget.
• Start with common LoRA settings (rank 8 to 16, alpha around twice the rank) and change one thing at a time.
• Log every run: dataset version, base model, settings, training loss and evaluation scores.
• Train for a small number of epochs first; overfitting shows up as falling training loss with worse evaluation results.
• Save the adapter, the config and the exact data hash together.

Loss curves tell you training ran, not that the model is useful. Score outputs on the held-out set with criteria tied to the task: exact match for classification, field accuracy for extraction, and a rubric for writing tasks. For writing, have a person rate a sample blind, without knowing which output came from the tuned model and which from the baseline.

Also test for regressions. A model tuned hard on product descriptions may get worse at following unrelated instructions. Keep a small general test set and check it on every run.

Serving frameworks such as vLLM can load several LoRA adapters on one base model and choose the adapter per request. That keeps costs down when you run separate adapters per brand or per task. Merging the adapter into the base weights is another option when you need one standalone model.

Maintenance is mostly data maintenance. When the house style changes, update examples and retrain. Schedule a quarterly review where you compare the adapter against the newest base models; sometimes a newer general model with a good prompt beats an older tuned one.

The biggest one is training before the evaluation set exists. Without fixed test examples, every run looks promising and nobody can say which is better. The second is changing several settings at once, which makes it impossible to learn what helped. The third is chasing tiny loss improvements while ignoring obvious data problems, such as duplicated examples or outputs that contradict each other.

Infrastructure also eats time. Pin library versions, keep a container image for training and write a short README for each project. The next person, often you three months later, will thank you.