Multimodal AI refers to models that understand and produce more than one type of data, such as text, images, audio and video, within a single system. Examples include GPT-4o, Gemini and Claude with vision input. Proving their value means measuring task accuracy on real inputs, human review time, cost per output and the business result the task supports, not how impressive a demo looks.
What counts as multimodal AI in marketing work?
Most current frontier models accept images alongside text, and several handle audio and video. In marketing, common tasks include reading screenshots of ad dashboards, writing alt text and product descriptions from photos, checking creative against brand rules, transcribing and summarising video, and extracting data from PDFs or scanned documents.
Generation is multimodal too: image models, voice synthesis and video generation. Measuring generation and understanding needs slightly different metrics, so separate them from the start.
Why is measurement harder than with text-only AI?
Text outputs can be compared with reference answers fairly easily. Multimodal tasks often lack a single right answer. Two good alt texts for the same photo may share few words. An image that follows the brand guide can still miss the creative intent.
Inputs vary more as well. Blurry photos, unusual angles, dense charts and accented speech all affect accuracy. A model that does well on clean examples may struggle on the messy files your team actually uploads.
Which KPIs should you track?
Choose a few metrics per task family and keep them stable over time.
For understanding tasks
- Accuracy against a reviewed sample, scored with a rubric such as correct, minor edit, major edit, wrong.
- Extraction accuracy at field level for documents: share of fields correct.
- Hallucination rate: share of outputs describing something not present in the input.
- Human review time per item, measured with a timer during spot checks.
For generation tasks
- First-pass approval rate: share of outputs approved without changes.
- Iterations per approved asset.
- Brand compliance issues found per batch.
- Performance of published assets, such as click-through rate in ads, against the previous method.
Cost and speed
Track cost per completed item, including retries, and turnaround from request to approved output. Images and audio can cost far more per call than text, so cost per item often surprises teams.
How do you build a test set for images and video?
Collect real inputs, not stock examples. A good test set includes the easy cases, the typical ones and the awkward ones. For a product description task, that means studio shots, user photos, items with text on packaging and items photographed against busy backgrounds.
Aim for at least 100 items per task. Have two people agree on what an acceptable output contains, and write it down as a checklist rather than a perfect answer. Checklists suit multimodal tasks better than exact references.
What should the dashboard show?
Keep it to one screen per task family:
- Weekly accuracy or first-pass approval trend.
- Hallucination count, with links to examples.
- Median human review time per item.
- Cost per item and total spend.
- Volume processed.
- One business metric, such as product pages published per week or ad variants tested.
Links to examples matter. A number alone does not tell a reviewer what failed; three bad examples usually do. Store those examples in a shared folder with the date and model version, so the history of failures stays available after the dashboard numbers move on.
How do you compare models fairly?
Run the same test set through each candidate with the same instructions. Score blind if you can, with reviewers not knowing which model produced which output. Note that image resolution settings, which some APIs expose, change both accuracy and cost.
Repeat the comparison when models update. Vendors release new versions often, and the ranking between them on your specific task can change.
How Alien Road applies multimodal AI measurement
In a typical engagement, discovery lists every task where staff look at images, documents, audio or video and write something about it. Product catalogue descriptions, creative checks against brand guides and video transcripts are frequent candidates.
Setup builds a test set for each chosen task from the client’s own files, defines the scoring checklist with their team, and compares two or three models on it. The chosen workflow runs with human review on every output at first, and spot checks later once accuracy is stable.
Measurement uses the dashboard above. For example, a monthly report might show review time per product description, first-pass approval rate and cost per description, next to the number of product pages published. When accuracy dips, the report links the failed examples so the client can see the cause, often a new type of input the test set did not include.
Which pitfalls distort the numbers?
Watch for these:
- Testing on clean samples and deploying on messy uploads.
- Counting time saved without counting review time.
- Ignoring hallucinations because the output reads well.
- Changing prompts and models at the same time, so you cannot tell which helped.
- Measuring output volume without a business outcome attached.
Who should own the measurement?
Assign one person per task family to own the numbers. Their job is small but regular: run the weekly spot check, update the test set when a new failure type appears, and note any prompt or model change in a change log next to the dashboard.
Without a named owner, the dashboard is built once and abandoned. The change log is especially useful. When accuracy drops, the first question is always what changed, and a dated list of edits answers it quickly. It also helps when presenting results to leadership, because you can show which adjustments improved accuracy and which ones did not.
Where to start this week
Choose one task where your team describes images or documents in writing, such as alt text for a product catalogue. Pull 100 real items, write a short acceptance checklist, and run them through two multimodal models. Time how long a reviewer takes to approve or fix each output. Accuracy, review time and cost per item from that one test will tell you whether a wider rollout is worth it.
Keep reading: AI Optimization · AI Optimization