Products Solutions Templates Pricing Resources

Incrementality Testing: Measuring Whether a Channel Truly Drives Results

Home / Everything About / Everything About Analytics / Incrementality Testing: Measuring Whether a Channel Truly Drives Results

Attribution tells you which channel gets credit for a conversion. But does the channel actually drive the conversion, or would it have happened anyway? Incrementality testing answers this question by running experiments: expose some users to the channel (treatment group), don't expose others (control group), measure the difference in conversion rates. The difference is the true incremental impact. This chapter covers incrementality testing and when to use it.

Attribution vs. Incrementality

Attribution: of people who converted, which channel touched them last? Simple to measure, but doesn't prove causation.

Incrementality: if we remove this channel entirely, how much revenue would we lose? This proves causation and measures true impact.

Example: 100 people converted. Google Ads last-click = 60 get credit. But incrementality test shows: if we didn't run Google Ads, 50 of those 60 would have still converted (through other channels). True incrementality of Google Ads = 10 conversions, not 60.

Types of Incrementality Tests

Geo-based testing

Run ads in some geographies (treatment), not others (control). Compare conversion rates. Example: run Google Ads in San Francisco (treatment), not in Berkeley (control). If SF has higher conversion rate, ads are incremental.

Advantage: simple, fast. Disadvantage: geography differences may confound results (SF is wealthier, skews conversion rate higher).

Holdout groups

Run ads to everyone except a random 10% (control). Compare conversion rates: ad-exposed users vs. holdout users. Difference is incrementality.

Advantage: accurate, controls for confounds. Disadvantage: requires large budget (we're not targeting 10% of our audience to measure impact).

Time-based testing

Run ads for some time periods (treatment), not others (control). Example: run paid search Mon-Thurs, not Fri-Sun. Compare conversion rates. If Mon-Thurs is higher, ads are incremental.

Advantage: simple. Disadvantage: day-of-week confounds (weekends are naturally lower conversion).

Running an Incrementality Test

Step 1: Define hypothesis. "Google Ads drives incremental conversions beyond baseline."

Step 2: Choose test type. Holdout groups is most accurate; use that if budget allows.

Step 3: Size your test. Need enough sample to detect difference. If baseline conversion = 5%, and you want to detect 2% lift, you need ~10,000 per group. Consult statistician or use power calculator.

Step 4: Run test for 2-4 weeks. Longer tests reduce variance from daily fluctuations.

Step 5: Analyze results. Is difference statistically significant? (p < 0.05). If yes, you've proven incrementality.

Challenges in Incrementality Testing

Cost: holdout groups mean losing revenue on purpose (we don't target 10% of our audience). Only worth it for high-spend channels where the insight is valuable.

Cannibalization: if user doesn't see your ad, they may use organic search or direct traffic instead. Incrementality captures this (it's included in control group behavior).

Long-term effects: ads may have brand effect that shows up days or weeks later. Most tests are short-term; miss long-term impacts.