Incrementality Testing: Proving Which Ads Actually Work

In the digital space, incrementality refers to the lift in a desired outcome, such as sales or lead submissions, that is directly caused by a specific marketing intervention. Most standard attribution models, like last-click or even multi-touch attribution, simply assign credit to the last touchpoint before a conversion. This often leads to a situation where a brand pays for a customer who was already planning to buy from them anyway.

For example, a user might search for your brand name on Google, click a branded search ad, and complete a purchase. A last-click model gives 100% credit to that ad. However, if that user would have clicked the organic search result right below it and bought the same product, the ad was not incremental. Incrementality testing helps you spot these redundancies and reallocate budget to more effective areas.

Marketers generally have two primary ways to structure these tests. The first is a geo holdout, which involves selecting specific geographic regions—like certain cities or states—to serve as a control group where ads are paused or restricted. The second is an audience holdout, where the advertising platform randomly splits your target audience into two groups regardless of their location, holding one back from seeing the ads.

Geo holdouts are useful when you want to measure the total effect of a media mix across all channels, including offline ones. They are less susceptible to cookie deletion or privacy-related tracking issues. However, they require careful matching of “twin” markets to ensure that the control city behaves similarly to the treatment city in terms of demographics and historical sales volume. If you test in a city with a unique seasonal trend, your results may be skewed.

Audience holdouts are typically easier to execute within a single platform like Meta or Google. They provide high granularity but are limited by the platform’s ability to track users consistently. If a user switches devices or clears their browser data, they might accidentally see an ad despite being in the holdout group. This contamination can make the ads seem less effective than they actually are.

Most major platforms offer built-in tools to conduct these tests without requiring complex external data science. On Meta, you can find the “Experiments” tool within the Ads Manager menu. This allows you to set up a Conversion Lift study where the platform automatically manages the split and provides a report on the incremental conversions and incremental ROAS (iROAS).

Google Ads offers similar functionality through its lift measurement suite. This is common for YouTube and Video campaigns but can also be applied to Search and Discovery in specific account configurations. These platform-native tools are a great starting point for brands spending at least $20,000 to $50,000 per month on a single channel. This level of spend is usually necessary to generate enough conversion volume for statistical significance.

Marketing Mix Modeling (MMM) is a top-down statistical method used to estimate the impact of various marketing channels on sales. While MMM is excellent for long-term planning, its results are based on historical correlations which can sometimes be misleading. By running regular incrementality tests, you can provide the MMM with “ground truth” data points to ensure the model is accurate.

If your MMM suggests that your social media ads have a 4x ROI, but a conversion lift study shows an incremental ROI of only 1.5x, you can use that experiment result to calibrate the model. This ensures that the weights and coefficients in your MMM reflect actual causality rather than just coincidental timing. Alien Road practitioners often use this hybrid approach to provide a more realistic view of budget allocation.

A successful test requires a clear hypothesis and a strict control over external variables. Start by defining what you want to measure. For instance, you might ask: “Does increasing our YouTube budget by 20% in the Midwest lead to a proportional increase in new customer acquisitions?” Once the question is set, you must ensure that no other major changes occur during the testing period.

• Define your success metric: Focus on a bottom-funnel metric like sales or qualified leads rather than clicks.
• Select a representative time frame: Tests usually need to run for at least 14 to 30 days to account for weekly shopping cycles.
• Ensure sufficient sample size: You need enough conversion events in both groups to be 95% confident in the results.
• Avoid overlapping changes: Do not launch new sitewide promotions or change your landing pages during the test.
• Isolate the channel: Ensure that other marketing activities remain constant across both the treatment and control groups.

Retargeting is often the channel with the highest reported ROAS but the lowest incrementality. Since you are targeting people who have already visited your site, many of them would have returned to complete their purchase anyway. A common pattern in a typical account is seeing a 10x ROAS on retargeting but finding that the actual lift is closer to 1.1x when a holdout is applied.

To test this, you can create a specific audience holdout for your retargeting campaigns. If the group that sees the retargeting ads does not convert at a significantly higher rate than the group that does not, you are likely overspending on that channel. Reallocating that budget to top-of-funnel prospecting can often lead to better long-term growth for the brand.