Small language models are compact AI models, usually with a few billion parameters or less, that can run on a laptop, phone or modest server. Their main risks for a marketing team are lower accuracy on complex tasks, confident errors in facts and languages they handle poorly, and unmanaged copies spreading across devices. Governance should limit them to narrow, testable tasks, require evaluation before use and track where each model runs.
Examples include Microsoft’s Phi family, Google’s Gemma, Meta’s smaller Llama variants, Mistral’s small models, Alibaba’s Qwen small versions and models that run inside phones such as Gemini Nano and Apple’s on-device foundation model. Tools like Ollama, LM Studio and llama.cpp make it easy to download and run them on a personal computer.
Their appeal is clear: low cost per task, fast responses, the option to keep data entirely in-house and the ability to fine-tune them cheaply on a specific job. These advantages are real. So are the limits.
Where are small models strong?
Small models do well on tasks that are narrow and well defined. Classifying support messages into categories, extracting fields from forms, tagging product reviews by topic, rewriting short text in a fixed style and simple routing decisions are all good examples.
They also do well when fine-tuned. A small model trained on a few thousand examples of a specific task can match or beat a large general model on that task, at a fraction of the running cost.
What are their limits?
Reasoning and long context
Small models struggle with multi-step reasoning, long documents and tasks that require holding many facts at once. Asking one to analyse a campaign report and recommend budget changes usually produces shallow or wrong advice.
Knowledge and facts
With fewer parameters, they store less world knowledge. They are more likely to invent facts, names and figures, especially about niche topics, and to do so fluently.
Languages
Many small models are trained mostly on English. Quality in Turkish, Azerbaijani or other languages may be noticeably lower, with grammar errors or literal translations. Test each language separately.
Safety tuning
Small models may have weaker safety and refusal tuning than large commercial models. They can produce inappropriate content more easily if prompted badly or by a malicious user.
Which risks need governance?
Shadow deployments
Because anyone can download a model and run it on a laptop, teams may start using them without IT knowing. Customer data pasted into a local model may stay private, but outputs may be used without review, and nobody knows which model version produced what.
Licence terms
Small models come under different licences. Some are permissive like Apache 2.0, others have custom terms with use restrictions. Using a model commercially without checking its licence is a legal risk.
Unvalidated outputs
A small model that classifies leads or support tickets can misroute a share of them silently. Without an evaluation set and monitoring, errors only show up as complaints.
Model supply chain
Models downloaded from public hubs may be modified versions. Use official sources and verify publishers. Model files in some older formats can even carry executable code, which is why safer formats like safetensors are preferred.
Which governance rules should come first?
- Approved list: name the models allowed for business use, their official source and licence.
- Use-case approval: each new use case is described in one paragraph and approved by an owner.
- Evaluation before use: test each model on at least fifty real examples per task and language, with an acceptance threshold.
- No unreviewed customer-facing output: anything a customer sees is reviewed or limited to safe templates.
- Data rules: what data may be processed, where the model runs and how logs are kept.
- Version tracking: record the model name and version used for each workflow.
- Retest on change: when a model is updated or replaced, rerun the evaluation set.
When should you choose a larger model instead?
Choose a larger model when the task involves reasoning, long documents, open-ended writing that customers will read or many languages at high quality. A common design uses both: a small model for quick classification and routing, and a larger model for the cases the small one flags as uncertain.
Cost alone is a weak reason to choose a small model if errors create rework. Compare total cost including review and correction, not only the token price.
How Alien Road applies small language models
In a typical engagement small models appear in data and operations tasks rather than content writing. Discovery identifies high-volume, narrow tasks such as tagging reviews, sorting inbound messages or extracting fields from lead forms, and checks the languages involved.
Setup uses an approved model from an official source, run either through a managed host or on the client’s infrastructure with Ollama or vLLM. We build an evaluation set from the client’s own data, compare the small model with a larger one and record results. If the small model reaches the agreed accuracy, it goes into the workflow with a fallback to a larger model or a person for uncertain cases.
Measurement tracks accuracy on a weekly sample, fallback rate and cost per task. Reports show these over time and flag any drift, such as a drop in accuracy after a change in incoming data.
How do you monitor a small model after launch?
Evaluation before launch is not enough, because incoming data changes. A new product range, a campaign in a new language or a seasonal shift in customer questions can push a small model outside what it handles well. Sample a set of outputs every week and have a person grade them against the same standard used in the original evaluation.
Watch the fallback rate as well. If more cases are being passed to a larger model or a person, the small model is struggling, even if the graded sample looks fine.
Where to start this week
List every place your team currently uses AI and note which models are behind it, including anything running locally on laptops. You may find small models already in use without approval.
Then pick one narrow, high-volume task and build a fifty-example test set from real data. Run a small and a large model on it and compare accuracy and cost. That single comparison tells you whether a small model belongs in your workflow.
Keep reading: AI Optimization · AI Optimization