Small language models are compact AI models, typically with a few billion parameters or fewer, that handle narrow tasks such as classification, extraction and short rewriting at a fraction of the cost of large models. Their costs sit in hosting or low per-token API fees plus setup and evaluation. ROI comes from lower spend and faster responses on high-volume tasks, measured against a large-model baseline with the same quality bar.
What counts as a small language model?
There is no strict cut-off, but the term usually covers models that can run on a single GPU, a laptop or even a phone. Examples include Microsoft’s Phi family, Google’s Gemma models, smaller Llama variants from Meta, Mistral’s compact models and Qwen’s small versions. Cloud providers also sell small tiers of their own models, such as “mini”, “flash” or “haiku” variants.
The trade-off is clear: smaller models are cheaper and faster, but have less general knowledge and weaker reasoning on complex tasks. They shine when the task is well defined.
Which marketing tasks suit small models?
- Classifying support tickets, reviews or leads into fixed categories.
- Extracting fields from text, such as product attributes from supplier descriptions.
- Sentiment tagging on social comments.
- Short rewrites with strict rules, such as shortening product titles to a length limit.
- Routing: deciding which larger model or human queue should handle a request.
Tasks that need broad knowledge, long documents or careful reasoning, such as strategy summaries or long-form content, still suit larger models.
Where do the costs sit?
Option 1: hosted small models via API
Many providers offer small models through an API, billed per token at rates well below their flagship models. Setup is fast and there is no infrastructure to run.
Option 2: self-hosting open-weight models
Open-weight models can be run on your own servers or cloud GPUs with tools such as vLLM, Ollama or llama.cpp. You pay for compute whether or not the model is busy, plus engineering time for deployment, monitoring and updates. Self-hosting tends to pay off when volume is high and steady, or when data must stay inside your infrastructure.
Shared costs
- Evaluation: building a test set and comparing models.
- Prompt work or fine-tuning, which small models often need more than large ones.
- Monitoring for quality drift over time.
How do you build the budget?
Start with the task volume and the current cost, if any. For example, an e-commerce company might classify 50,000 customer reviews a month and currently send them all to a large model.
- Estimate tokens per item and items per month.
- Price the large-model baseline at current rates.
- Price two or three small-model options: hosted API and, if relevant, self-hosted compute.
- Add setup hours for evaluation and prompt tuning.
- Add a monthly allowance for monitoring and re-testing.
Calculate the break-even month for each option. Hosted small models often break even almost at once; self-hosting usually needs higher volume.
How do you prove the quality is good enough?
Cost savings mean nothing if output quality drops below what the business accepts. Build an evaluation set before choosing.
- Collect a few hundred real examples with correct answers agreed by a person.
- Run the large model and each small model on the same set.
- Score accuracy, format compliance and failure types.
- Set a minimum acceptable score with the business owner.
Often a small model reaches the bar on most categories but fails on a few. A common pattern is routing: the small model handles items it is confident about, and the rest go to a larger model or a person.
Which ROI metrics should you report?
- Cost per processed item, before and after.
- Monthly spend on the task.
- Accuracy against the evaluation set, re-checked monthly on fresh samples.
- Median response time, important for real-time features such as chat routing.
- Share of items escalated to a larger model or human.
Present cost and quality together. A saving that comes with a hidden error rate tends to come back as rework.
How Alien Road applies small language models in AI optimisation
A typical engagement starts with a review of the client’s current AI usage and bills. The team lists tasks by volume and complexity, and flags high-volume, narrow tasks as small-model candidates.
Setup involves building an evaluation set from the client’s real data, testing hosted small models first and self-hosted options only where volume or data rules justify them. Routing rules send uncertain items to a larger model.
Measurement is monthly: cost per item, accuracy on fresh samples, escalation share and response time. The report recommends whether to extend small models to another task or adjust the routing threshold. Model choices are revisited when providers release new small tiers, since the cheapest model that passes the evaluation set changes over time.
A short scenario: review tagging for a marketplace seller
Consider a marketplace seller that tags every incoming product review by theme: sizing, quality, delivery, packaging and price. It currently uses a large model for this, and the bill grows with review volume. The team labels four hundred reviews by hand and tests two hosted small models against the large one.
One small model matches the large model on four of the five themes but confuses packaging and delivery complaints. The seller adds a routing rule: reviews mentioning boxes, couriers or damage go to the large model, everything else to the small one. Cost per review falls sharply, accuracy on the test set stays above the agreed bar and the routing threshold is reviewed monthly.
Where to start this week
Find the single highest-volume AI task in your stack and export a sample of a few hundred inputs. Label the correct outputs by hand. Run that sample through a small hosted model and compare the results with your current model. If the small model is close, you have a strong case for a pilot; if not, you have learned that cheaply. Either way, keep the labelled sample. It becomes the evaluation set for every future model decision on that task, and it costs nothing to rerun when prices or models change.
Keep reading: AI Optimization · AI Optimization