Measuring Shopify Sidekick and Shopify Magic: KPIs That Prove Value

Free features still use people’s time and still shape what customers see. A store that publishes 300 generated descriptions without checking whether they convert could lose sales slowly and never notice. Measurement also helps decide where to invest saved time: if the team spends four fewer hours a week on copy, where did those hours go?

Start with time, because that is where the value usually appears first.

• Time per product listing, from attributes to published page.
• Time per email campaign, from brief to scheduled send.
• Time per image edit for background removal or replacement.
• Number of support or how-to questions answered by Sidekick instead of a colleague or the help centre.

Measure a baseline for two weeks before rollout, then repeat after a month. Record the task type, the product or campaign, start and end time, and whether AI was used. Four columns are enough, and keeping the log simple is what makes people fill it in. Self-reported time logs in a shared sheet are good enough for a small team.

Edit rate

Edit rate is the share of AI-generated text that reviewers change before publishing. A low rate means drafts are usable; a high rate means the feature saves little time. Track it per content type, because descriptions and email copy often behave differently.

Efficiency means little if results drop. Compare AI-assisted content against human-written content on:

• Product page conversion rate and add-to-cart rate, from Shopify Analytics.
• Organic sessions and impressions for product pages, from Google Search Console.
• Email open and click rates in Shopify Email or your email platform.
• Return rates and customer questions about products, which can rise when descriptions are inaccurate.

The last point is often forgotten. A description that oversells material quality or sizing can increase returns. Tag returns with a reason code so you can see patterns.

A simple controlled comparison works for most stores.

• Pick two groups of similar products, matched on price, traffic and category.
• Give one group AI-assisted descriptions (generated and edited) and keep the other as is.
• Leave other variables unchanged for four to six weeks: price, images, ads.
• Compare conversion, add-to-cart and organic sessions between groups.

For email, Shopify Email and most email tools allow A/B testing of subject lines. Test a human-written line against a generated one on the same send.

Seasonality can distort results, so avoid starting a comparison right before a major sale period such as Black Friday, when traffic and intent change sharply for every product.

Small stores will not reach statistical significance on every test. That is fine; look for consistent direction across several tests rather than one decisive result.

Keep it to one page with three sections:

• Efficiency: time per task, edit rate, number of AI-assisted assets this month.
• Outcomes: conversion and organic metrics for AI-assisted pages versus the comparison group.
• Adoption: which team members use the features and for what.

Shopify’s own reports cover sales and conversion; Search Console covers organic search; a spreadsheet covers time logs. Looker Studio can combine them if you want one view, but a shared sheet updated monthly is often enough.

Act on them. If edit rates on descriptions are high, improve the inputs: complete product attributes, add a short brand voice note to your prompts or keep hero products human-written. If AI-assisted pages convert as well as others and take less time, extend the approach to the rest of the catalogue. If return reasons point to inaccurate descriptions, tighten review before anything else.

Avoid vanity measures

Counting the number of descriptions generated or Sidekick questions asked says little on its own. Those are activity figures. A store can generate 500 descriptions and lose conversion at the same time. Always pair an activity count with a time measure and an outcome measure, so the report shows whether the activity produced anything useful.

Revisit the KPI list every quarter. Shopify updates these features often, and a feature that was weak six months ago may now be useful, or the reverse.