Fine-tuning and LoRA prove their value when a custom model beats a well-prompted base model on a fixed evaluation set, at an acceptable cost and latency, without losing general ability. The KPIs that show this are task quality against a held-out test set, human edit rate, cost per output, response time and regression on tasks the model already did well.
Fine-tuning means further training a pre-trained language or image model on your own examples so it learns a style, format or task. LoRA, short for low-rank adaptation, is a parameter-efficient method that trains small adapter matrices instead of all the model’s weights, which makes training cheaper and lets you swap adapters for different tasks. Many teams use LoRA on open-weight models, while hosted providers offer managed fine-tuning on some of their models.
Why is measurement harder than training?
Training a LoRA adapter can take hours. Knowing whether it helped takes planning. Generated text has no single right answer, so “accuracy” needs a definition. And the comparison point matters: a fine-tuned model is only worth maintaining if it beats the cheaper option of a good prompt with a few examples, or retrieval from your own documents.
Without a baseline, teams often judge a custom model on a handful of impressive outputs. That is how projects get approved and later quietly abandoned.
Step 1: build an evaluation set before training
Set aside examples the model never sees in training. For a brand-voice copywriting model, that might be a few hundred product briefs with approved human-written copy. For a classification task, such as sorting support tickets, it is labelled tickets across all categories, including rare ones.
- Keep the test set separate from training data, with no near-duplicates.
- Cover edge cases: short inputs, long inputs, mixed languages, unusual products.
- Freeze it. Changing the test set mid-project makes earlier scores meaningless.
- Record the base model, prompt and settings used for the baseline run.
Which quality KPIs matter?
The right metric depends on the task. For classification and extraction, use precision, recall and F1 per category, not just overall accuracy, because a model can score well while failing every rare class. For structured outputs, measure the share of responses that parse and pass validation.
For writing tasks, combine three views. Human ratings on a simple scale for accuracy, brand fit and usefulness. Pairwise preference, where reviewers choose between base and fine-tuned outputs without knowing which is which. And edit rate in production: how much of each output a person changes before publishing. Edit rate is often the most convincing number for stakeholders because it maps to time.
Model-graded evaluation, where a strong model scores outputs against a rubric, scales well. Calibrate it against human ratings on a sample first; if the grader disagrees with people often, do not trust it alone.
Which cost and speed KPIs matter?
- Cost per output: inference cost for the custom model against the base model with a long prompt.
- Training and retraining cost, including data preparation time.
- Latency: median and slow-end response times, since users notice the slow cases.
- Throughput under load if the model serves many users.
- Maintenance hours per month for monitoring and updates.
A fine-tuned smaller model sometimes wins on cost because it needs a shorter prompt and less compute than a large general model. That saving only counts if quality holds.
How do you check for regression?
Fine-tuning can make a model worse at things it used to do well. A model tuned heavily on short ad headlines may become poor at following longer instructions, or lose safety behaviour. Keep a small general test set, such as instruction following, factual questions in your domain and refusal of clearly inappropriate requests, and run it after every training round.
With LoRA, regression is easier to manage because the base model is untouched; removing the adapter restores original behaviour. That is a practical advantage worth noting in any business case.
What should the dashboard show?
One page, updated after each training round and weekly in production.
- Quality score on the frozen test set: base, previous version and current version.
- Pairwise win rate against the base model.
- Production edit rate and its trend.
- Cost per thousand outputs and median latency.
- Regression test pass rate.
- Volume: outputs generated and share accepted without edits.
- Notes on data changes, so score movements can be explained.
Tools such as MLflow or Weights and Biases track training runs well, while a simple spreadsheet or Looker Studio report is often enough for the business view.
When is fine-tuning not worth it?
If a prompt with a few good examples gets within a small margin of the fine-tuned model, the custom model rarely justifies its upkeep. If the task depends on facts that change often, such as prices or stock, retrieval is the better tool, because training bakes in information that goes stale. And if there are too few quality examples, the model learns noise. Fine-tuning shines when the task is stable, the format is specific and good examples are plentiful.
How Alien Road applies fine-tuning and LoRA
In a typical engagement, discovery checks whether fine-tuning is the right answer at all. We review the task, the volume, the available examples and how the client currently measures quality. Often we start with prompting and retrieval, and only propose LoRA or managed fine-tuning when the baseline shows a clear gap.
Setup includes building and freezing the evaluation set with the client’s reviewers, running the baseline, then training adapters on cleaned data. Each training round is scored on quality, cost, latency and regression before anything reaches production.
Measurement continues after launch through edit rates and periodic blind comparisons against the base model. Monthly reports use the dashboard above, so the client can see whether the custom model still earns its place as base models improve.
Where to start this week
Pick one task you are considering for fine-tuning and collect fifty examples of excellent output. Run them through your current model with a strong prompt, and have two people rate the results blind against the originals. That baseline tells you how much room a custom model has to improve before you spend anything on training.
Keep reading: AI Optimization · AI Optimization