On-device AI refers to models that run directly on a phone, tablet or laptop instead of in a remote data centre. Apple Intelligence on recent iPhones, iPads and Macs, and Gemini Nano on Pixel and other Android devices, summarise notifications, rewrite text, sort messages and answer some questions locally. For marketers the question is how to measure their effect, because these features change how people see emails, notifications and web content before anyone clicks.
What do Apple Intelligence and Gemini Nano do?
Apple Intelligence combines a small on-device model with Private Cloud Compute, Apple’s server system for heavier tasks, and can hand some requests to ChatGPT with the user’s permission. Features include writing tools, notification and email summaries, priority messages in Mail, Safari page summaries, image tools and an updated Siri.
Gemini Nano is Google’s smallest Gemini model, built for devices. On supported Android phones it powers features such as call and recording summaries, smart replies, scam detection and some accessibility tools. Google has also made a version available inside the Chrome browser for developers to test built-in AI APIs.
Both systems vary by device, language and region at the time of writing, which matters for any measurement plan.
Why does on-device AI matter for marketing?
These features sit between your message and the person reading it. A notification summary may compress your push message into a few words. Mail may move a promotional email out of a priority view, or show a summary instead of your preview text. Safari can summarise a landing page before the visitor scrolls. Smart reply suggestions change how quickly and how people answer messages from businesses.
None of this is under the marketer’s direct control, and most of it leaves no trace in analytics. That is why measurement has to rely on comparisons and proxies.
Which KPIs show the effect on email and notifications?
The aim is to spot changes in behaviour among users on devices with these features, compared with users on devices without them.
- Open and click rates by device and OS version, comparing devices that support Apple Intelligence with older models.
- Click-to-open rate, which shows whether the people who open still engage with the content.
- Push notification open rate by OS version, before and after major feature rollouts.
- Unsubscribe and complaint rates, which can rise if summaries make messages look less relevant.
- Conversion rate per email session by device class.
Keep in mind that Apple Mail Privacy Protection already inflates open rates on Apple devices by preloading images. Clicks and conversions are more reliable than opens for any Apple-related comparison.
Which KPIs show the effect on websites?
Page summaries and assistants reading pages may change on-site behaviour:
- Scroll depth and engaged time on long articles, by browser and device.
- Bounce rate for Safari users compared with other browsers, around feature release dates.
- Conversion rate on landing pages whose key facts sit in the first screen versus further down.
- Search Console clicks and impressions by device, to separate search effects from on-device effects.
Expect noise. Seasonal shifts, campaigns and site changes all move these numbers. Look for consistent gaps between device groups rather than single spikes.
What should the dashboard look like?
A focused dashboard needs only a few views.
Device segment view
Group sessions and email clicks into “on-device AI capable” and “not capable” based on device model and OS version. Show trends for each group on the same chart.
Message performance view
For email and push, show click-through and conversion by segment, with annotations for OS releases and feature launches.
Content structure view
For key landing pages, show engaged time and conversion alongside a note of whether the page states its main offer in the first paragraph. This helps test whether answer-first pages hold up better when summaries are common.
How Alien Road applies on-device AI measurement for clients
In a typical engagement the work starts with discovery of the client’s device mix. The team pulls the share of sessions, email clicks and app users on devices that support Apple Intelligence or Gemini Nano. For many consumer brands in Turkey and Europe that share is growing but still a minority, which shapes how much effort the topic deserves.
Setup involves building the device segments in GA4 or the client’s analytics tool and the email platform, plus annotations for major OS releases. Where the client sends push notifications, message templates are reviewed so the first words carry the key information, since summaries tend to keep the start.
Reporting adds a short monthly section to the regular performance report: segment comparisons, any widening gaps and suggested tests. Typical tests include subject lines that state the offer plainly, preview text written as a one-line summary and landing pages that answer the main question in the opening paragraph.
How should content and messages adapt?
Most practical advice follows from how summarisers work. They pick out the main point and drop the rest. Messages that lead with the point survive compression better.
- Put the offer or key fact in the first line of emails and notifications.
- Keep subject lines literal rather than clever; summaries tend to strip wordplay.
- Use clear headings and short paragraphs on landing pages so page summaries capture the right content.
- Avoid hiding prices or deadlines in images, which summarisers may skip.
What are the limits of this measurement?
Analytics cannot see whether a user read a summary instead of your full message. Device capability does not mean the user turned the features on. Language support differs, so Turkish-speaking users may not have the same features as English-speaking users on the same phone. Treat results as direction, not proof, and combine them with simple controlled tests where possible.
Where to start this week
Export last month’s email clicks and website sessions by device model. Mark which devices support Apple Intelligence or Gemini Nano and calculate their share. If it is above a few percent of your audience, create the two segments in your analytics tool and annotate the next OS release date, so you have a clean before and after when it arrives.
Keep reading: AI Optimization · AI Optimization