An order arrives at 9.14 on a Thursday evening. The customer opened an email that morning, clicked a paid-search result two days earlier, visited through a social post the week before and had already bought from the retailer three times. Four channels can show evidence that they were present. None can prove that the order depended on them.
That distinction is the quiet fault line beneath ecommerce reporting. Attribution allocates credit among recorded interactions. Incrementality asks whether the intervention created an outcome that would otherwise not have occurred.
The difference matters because an attributed sale and an additional sale are not the same commercial event. A loyal customer who was already going to replenish may click an email on the way to checkout. A branded-search advert may sit above the organic result the customer intended to use. A discount may claim a conversion while merely reducing the margin on an order that was already coming.
None of those reports is necessarily false. Each is answering a narrower question than the one the board needs answered.
Attribution describes the route; incrementality tests the cause
An attribution model assigns some or all of a conversion to the touchpoints recorded before it. Last click, first click, linear, time-decay and data-driven models distribute the credit differently, as our guide to ecommerce attribution models explains. They can help describe how customers moved through a journey and how channels overlapped.
They cannot, by themselves, reveal the counterfactual: what would this same population have done if the marketing activity had not occurred?
That question cannot be answered by giving the sale a different label. It requires a comparison. A properly designed experiment divides an eligible population into a treatment group, which receives the intervention, and a control or holdout group, which does not. The commercial effect is the difference in outcomes between them, subject to uncertainty and the quality of the design.
Google describes the same principle in its documentation for lift studies: treatment and control groups are separated so that the difference in their aggregate behaviour can be used to estimate the lift caused by advertising. This is causal measurement rather than a contest for credit.
Start with the decision, not the dashboard
An incrementality test should begin with a decision that somebody is prepared to make. “How much revenue did email produce?” is usually too vague. Better questions are:
- Does sending this autonomous product-selection programme create more net revenue than sending nothing to the same eligible population?
- Does a promotion generate enough additional gross profit to pay for the discount and media cost?
- Does increasing paid-social investment create new demand, or capture customers already on their way?
- Does suppressing habitual purchasers from an incentive preserve sales while improving margin?
This discipline prevents a test becoming an elaborate way of confirming a preferred story. It also determines the experimental unit, eligibility rules, duration and primary measure.
Revenue is often the right primary outcome, but not always. For low-margin categories, incremental gross profit is more useful. For subscription retail, the relevant measure may be incremental contribution over a defined period. For replenishment businesses, purchase frequency and the timing of the next order may matter alongside immediate sales.
Select one primary outcome before the test starts. Secondary measures can explain what happened, but changing the definition of success after seeing the result converts analysis into negotiation.
Build a holdout that can stand in for “what would have happened”
The control group is not a collection of people the business considers unimportant. It is the estimate of the world without the intervention. If that estimate is weak, the answer is weak.
Random assignment is valuable because it gives every eligible customer the same probability of entering treatment or control. With sufficient volume, known and unknown differences are more likely to be balanced across the groups. Historic spend, category preference, loyalty, recency and propensity should not determine who is deliberately placed on the more convenient side.
The eligibility rules must also be identical. If treatment includes all contactable customers but control excludes recent purchasers, the two groups no longer answer the same question. Likewise, removing high-value customers from the holdout because their revenue feels too precious makes the experiment safer emotionally and poorer statistically.
For individualised email, the cleanest test is often customer-level randomisation performed before product selection and send-time decisions. The treatment group proceeds through the normal system. The holdout group receives no intervention from the programme being tested. Orders from both groups are then measured over the same window.
The percentage held out should be large enough to detect a commercially meaningful difference without withholding an excessive number of opportunities. There is no universal percentage. It depends on audience size, normal purchase rate, revenue variability, expected lift and the level of confidence required. A small retailer may need a longer test; a large retailer may reach a useful answer quickly with a modest holdout.
Measure the difference in rates, not the pile of attributed orders
Suppose 95,000 eligible customers receive the treatment and 5,000 enter the holdout. During the measurement window:
- treatment revenue is £570,000, or £6.00 per eligible customer;
- holdout revenue is £27,000, or £5.40 per eligible customer.
The observed incremental revenue per eligible customer is £0.60. Applied to the 95,000 treated customers, the estimated incremental revenue is £57,000.
The programme may have been present before substantially more than £57,000 of sales. That larger attributed figure describes association. The £57,000 estimate addresses causation.
The same structure works for conversion:
Incremental conversion rate = treatment conversion rate − control conversion rate
For investment decisions, calculate incremental return on ad spend or programme cost:
Incremental ROAS = incremental revenue ÷ incremental cost
Where margin varies materially, substitute incremental gross profit for revenue. A campaign can produce positive incremental revenue and still destroy profit through discounting, fulfilment, returns or media cost.
Google’s explanation of Conversion Lift reporting makes the same useful distinction: incremental conversions are the difference between treatment and control outcomes, whereas attributed conversions are those assigned to ads by the platform’s attribution system. Reporting both is informative; confusing them is expensive.
Do not declare victory without uncertainty
The difference between two groups is an estimate, not an oracle. Ordinary variation can produce a gap even when the intervention has no effect. A credible report therefore includes:
- the size of the treatment and control groups;
- the observation window and any lag allowed for conversion;
- revenue or profit per eligible customer in both groups;
- the estimated incremental effect;
- a confidence interval or equivalent expression of uncertainty;
- exclusions, data losses and deviations from the original design.
A result of “£57,000 incremental revenue” is incomplete if the plausible range is minus £20,000 to plus £134,000. The point estimate may look attractive while the evidence remains inconclusive.
This is particularly important in digital marketing, where the true effect is often small relative to natural variation in purchasing. A large comparison of 15 Facebook advertising experiments found that observational methods frequently failed to reproduce the causal effects obtained from randomised experiments, even with extensive demographic and behavioural data. The study, published in Marketing Science, is a useful warning against treating abundant tracking data as a substitute for experimental identification: A Comparison of Approaches to Advertising Measurement.
Before launch, estimate the sample size required to detect the smallest effect worth acting upon. If the business would only change course for a 5% lift, the test should be capable of detecting something near that threshold. Otherwise, the likely outcome is a costly “no clear result” followed by whichever decision people preferred beforehand.
Protect the experiment from contamination
Holdout testing fails quietly when the control group receives the treatment by another route. A customer withheld from one email may still receive a near-identical triggered campaign, a discount from another team or retargeting built from the same behaviour. Treatment customers may be excluded by deliverability rules that do not apply to control. Orders may be counted differently across channels.
Create an exposure map before the test. Record which communications remain permitted for both groups, which intervention is being isolated and where suppression must occur. Keep operational messages, such as receipts and service notices, separate from commercial treatment.
Conversion windows also need restraint. Too short, and delayed purchases are missed. Too long, and unrelated activity adds noise. Review the normal interval between exposure and purchase, then set the window before looking at the result. Our examination of sustaining email attribution models provides the broader context for overlapping touchpoints and delayed conversion.
Novelty, seasonality and stock availability can distort a single test. Repeat measurement across materially different trading periods. A result during Black Friday may not describe an ordinary February; a result during a stock shortage may measure availability more than persuasion.
Use incrementality to govern the system, not decorate a report
The strongest measurement loop changes what happens next. If a programme repeatedly generates additional gross profit among lapsed customers but little among habitual replenishment buyers, the system should respond to that evidence. It might alter eligibility, contact pressure, product choice or the point at which intervention is useful.
This does not mean every individual decision should be manually rewritten after a test. An autonomous system should continue making individual product and timing decisions, while its commercial value is assessed against a stable holdout. The holdout judges the system’s effect; it should not be used to impose crude campaign rules that erase individualisation.
That separation is important. SwiftERM’s autonomous product selection concerns what each individual consumer is most likely to buy next. Incrementality concerns whether acting on those predictions creates commercial value beyond the orders that would have arrived anyway. Our analysis of autonomous hyper-personalisation economics describes the cost and operating context; a holdout supplies the causal proof.
Maintain a simple decision record for every completed test:
- What intervention was tested?
- Who was eligible?
- How were treatment and control assigned?
- What was the primary commercial outcome?
- What incremental effect and uncertainty were observed?
- What will change because of the result?
- When will the conclusion be tested again?
The final two questions prevent incrementality becoming another dashboard that everybody admires and nobody obeys.
The commercial answer attribution cannot supply
Attribution remains useful. Retailers need to understand journeys, channel overlap and the sequence of interactions before purchase. But attribution cannot establish that the marketing created the sale simply because it found a touchpoint.
Incrementality accepts a more demanding standard. It leaves some customers untreated, records what they do, and compares their behaviour with an otherwise equivalent treated population. It allows for uncertainty. It protects the design from contamination. Most importantly, it measures the outcome the business can actually act upon: revenue or profit that would not have existed without the intervention.
That figure is usually smaller than attributed revenue. It is also considerably more valuable.
Questions ecommerce teams commonly ask
Is incrementality the same as attribution?
No. Attribution distributes credit for observed conversions among recorded interactions. Incrementality estimates how many conversions, how much revenue or how much profit the intervention caused above a credible baseline.
Does a holdout mean deliberately losing sales?
It means withholding a defined intervention from a small, properly selected comparison group so the retailer can estimate its true effect. Without that comparison, the business risks spending far more on activity that receives credit without creating demand.
How often should an incrementality test be repeated?
Repeat it when the audience, intervention, trading conditions or underlying system changes materially, and periodically for always-on activity. A result is evidence about a defined treatment in a defined period, not a permanent law.
Further accreditation and references
- IAB and IAB Europe, Guidelines for Incremental Measurement in Commerce Media — counterfactuals, bias control and separation of signal from noise.
- IAB and Media Rating Council, Retail Media Measurement Guidelines — test-and-control design, transparency and measurement practice.
- Google Research, Measuring Ad Effectiveness Using Geo Experiments — randomised geographic treatment and control methodology.
- Google Research, Robust Causal Inference for Incremental Return on Ad Spend — estimating iROAS under heterogeneous geographic conditions.
- Liu, Bettaney and Chamberlain, Designing Experiments to Measure Incrementality on Facebook — statistical power and sample-size considerations for lift studies.
- Kerman, Wang and Vaver, Estimating Ad Effectiveness Using Geo Experiments in a Time-Based Regression Framework — counterfactual market response and cumulative causal effect.
