Small language models are AI models with far fewer parameters than frontier systems, typically a few billion rather than hundreds of billions, which lets them run quickly on modest servers, laptops or even phones. Examples include Microsoft’s Phi family, Google’s Gemma, Meta’s smaller Llama models, Mistral’s small models and Qwen. For e-commerce brands they suit high-volume, narrow jobs where speed, cost and data privacy matter more than broad knowledge.
What counts as a small language model?
There is no official cut-off, but the term usually covers models under roughly ten to fifteen billion parameters. Many are released as open weights, so you can host them yourself. Some, such as Gemini Nano and Apple’s on-device models, run directly on phones.
Smaller models know less about the world and handle long, complex reasoning less well. They make up for it with speed and cost. A small model can answer in a fraction of a second on hardware that costs little to run, which matters when you process millions of search queries or product records.
Which e-commerce tasks suit small models?
The pattern is the same across most good uses: clear input, limited set of possible outputs, and very high volume. When the task needs creative writing or wide general knowledge, a larger model is usually the better choice.
- Query understanding in site search: mapping “cheap black trainers size 42” to filters for colour, price and size.
- Product tagging: assigning categories and attributes from titles and descriptions.
- Review moderation: flagging spam, abuse or personal data before publishing.
- Support routing: classifying incoming messages by topic and urgency.
- Short text generation within strict templates, such as filter labels or snippet text.
- Spell correction and synonym handling for local languages.
Why choose a small model over a large one?
Cost at scale
Large hosted models charge per token. For a task that runs on every search or every new product, that adds up quickly. A self-hosted small model has a fixed server cost no matter how often it runs.
Speed
Search and autocomplete need answers in milliseconds. A small model on a nearby server can respond fast enough to sit inside the search box. A large model called over the internet often cannot.
Privacy
Customer messages and order data can stay on your own infrastructure. For brands handling data under GDPR or Turkey’s KVKK, that simplifies data processing agreements and transfer questions.
How do you make a small model good at your task?
Out of the box, a small model may be mediocre at your exact job. Three techniques close most of the gap, and they can be combined.
- Clear prompts with examples and a fixed output format, such as JSON with allowed values.
- Fine-tuning or LoRA on a few thousand of your own labelled examples.
- Distillation: using a large model to label data, then training the small model on those labels.
- Retrieval: giving the model your product data or help articles at answer time.
- Constrained decoding, where the inference engine only allows outputs from your list.
Distillation is especially practical. You pay a large model once to label a training set, check a sample by hand, and then run the cheap small model forever after.
What are the limits?
Small models make more factual errors and are more sensitive to how a prompt is worded. They can struggle with languages that were rare in their training data; test carefully in Turkish, Azerbaijani or Russian before relying on them.
Self-hosting also means you own uptime, updates and security. If no one on the team can maintain a server, use a managed host that offers small open models through an API, which still costs much less than frontier models for the same volume.
How Alien Road applies small language models
Our AI optimisation team looks for jobs where volume is high and the output is a label or a short structured answer. Search query parsing and product tagging are the most common starting points.
Setup usually follows a distillation route: a large model labels a sample, the client’s merchandisers correct it, and a small model is tuned on the corrected set. We host it where the client’s data rules require, often on an EU or local server.
Measurement compares the small model with the current method on accuracy, response time and monthly cost. For search, we also track zero-result rate and search exit rate before and after.
How do you test a small model before committing?
Build a test set of a few hundred real examples with correct answers, written or checked by someone who knows the catalogue. Run two or three small models and one large model on the same set. Record accuracy, speed per request and estimated monthly cost at your real volume.
If the best small model comes within a few points of the large one, the cost and speed advantage usually decide it. If the gap is wide, try fine-tuning before giving up, since a tuned small model often overtakes an untuned large one on a narrow task.
Can small models run inside the shopper’s device?
Yes, and this is growing. Browsers are adding built-in AI features, and phones from Apple, Google and Samsung run small models locally. For brands with mobile apps, on-device models can power features such as offline search suggestions or private chat without sending data to a server.
The trade-off is control. You cannot always choose which model runs on the device, and older phones may not support these features. Treat on-device AI as an extra layer, with a server-side fallback for everyone else.
Plan for the gap between pilot and production as well. A model that works in a notebook still needs monitoring, a fallback when the server is down and a way to roll back to the previous version. Simple logging of inputs, outputs and response times is usually enough to start.
Where to start this week
Export one week of site search queries and pick the hundred most common ones that returned zero or poor results. Write the correct filters for each by hand. Then run a small model such as Gemma or Phi on the same queries with a prompt that returns filters in JSON, and count how many it gets right. The score tells you whether a pilot is worth the setup.
Keep reading: AI Optimization · AI Optimization