Cyber Week 2025 set a record with $44.2 billion in online spending, according to Adobe. For the brands cited in MarTech’s piece, the five days from Thanksgiving through Cyber Monday produced 21.5% of all November and December revenue. When that much budget moves in a short window, giving a channel too much credit is an expensive mistake.
Yet the channels that report the biggest peak numbers are not always the ones creating the sale. The fix is to run holdout tests before peak spending begins, not after.
What a holdout test measures
A holdout test works as a controlled experiment. Pick one channel—a coupon partner, a content partner or another program—and temporarily switch it off for part of your audience while everyone else stays exposed. The difference in outcomes over the same period is the channel’s incremental contribution.
Incrementality is the part of a sale that would not have happened without that marketing touchpoint. If pausing a coupon partner leaves sales largely unchanged, the revenue it was credited with was probably going to arrive regardless.
Why last-click numbers mislead
Nielsen’s research found that 84% of teams say they can prove ROI, but only 38% examine every channel. That gap leaves room for over-crediting.
Coupon and loyalty partners are often over-credited at last click, so they make strong candidates for coupon holdouts. Creator marketing often happens off-pixel—on TikTok Shop or Amazon—so code or geo tests fit better.
The MarTech contributor gives a useful example: coupon partners may report 21x ROAS while content partners report 8x. If you prune by ROAS, content goes first. If you prune by new customer acquisition, content is often the last to go.
Design the test early
Geo holdouts, which pause a channel in matched regions, are the most common approach for affiliate programs. Audience or code holdouts, which suppress a coupon or link for a random share of a customer list, work for closed-loop programs.
For a practical test, the piece suggests holding out 20% to 30% of traffic, aiming for at least 200 conversions a week, and running for four to eight weeks. If a full holdout is too costly, start smaller: hold out a smaller segment or test one channel at a time.
- Switch off links and block coupon extensions to protect the control group.
- Inform key publishers and log promotions, flash sales and price changes.
- Watch for contamination, such as direct traffic growing in paused markets while brand search stays flat.
- Compare exposed and holdout audiences, then the previous period and the previous year.
Move decision-making from ROAS to incremental ROAS, or iROAS: test revenue minus control revenue, divided by spend, with groups scaled to equal size.
Timing and follow-through
Cyber 5 runs from November 26 to 30. A four-week test starting the week of October 26 ends around November 23, just before peak. Audit all links, UTMs and pixels before the test starts, because broken tracking cannot be fixed mid-test. Lock tags from November 1 through the holidays, and keep a single source of truth for every promotion, email and price change.
Pull results the week after Cyber 5, not in January. Before declaring success, check for pull-forward—buyers who would have purchased later anyway. Discounted volume from current customers mostly shifts the timing of revenue rather than creating new growth.
Then attach every finding to an owner, a budget line or a follow-up test. Put spend where there is proven lift, and consider commission structures that reward new customer acquisition.
Source: MarTech
