GuideMeasurement and attribution
Holdout test
Definition
An experiment that switches a channel’s spend off in matched regions and compares their sales with the regions left running. The gap between the two groups is the channel’s measured contribution.
How it works
Split your regions into two groups whose sales tracked each other before anyone touched anything; the match is most of the design. Switch the channel off in one group and run past your own buying cycle, because a shorter test ends before the missing spend would have surfaced as missing sales. The gap also has to clear the ordinary week-to-week swing in your numbers, which is what dictates how many regions and how many weeks are enough. And the test has a price: if the channel works, the dark regions lose real revenue for the duration. That cost is why holdouts get discussed and not run.
What to watch for
Chart both groups for the months before the start; if the lines were not tracking each other then, the gap during the test was there before the test. Confirm the channel is fully dark in the off group; a national campaign still delivering there shrinks the gap you are trying to read. Confirm nothing else moved in those regions either: a price change or a stockout lands in the result as the channel’s doing. And a result read before the buying cycle has cleared shows no loss, because the loss has not arrived yet.
The question you ask
“How much revenue are we prepared to lose to get this answer?”