Holdout testing, explained for Shopify brands
Every AI tool says it's working. A holdout is the simplest way to find out which ones really are. Here's how it works, how big it should be, and how to read the result.
- A holdout is a random slice of your audience that a tool never reaches. Comparing it with everyone else shows what the tool really added.
- Compare conversion rates, not totals. The groups are different sizes.
- Bigger holdouts give clearer answers but cost reach. 10–20% is a common range.
- Decide the test length up front and don't stop early because a number looks good.
What a holdout is
A holdout is a group of real shoppers you deliberately leave alone. Before a tool starts, you randomly set aside a share of your audience, say 15%. The tool can reach everyone else, but never that 15%.
Because the split is random, the two groups are alike in every way that matters except one: whether the tool reached them. So any difference in how often they buy is the tool's real effect. Marketers call that difference lift or incrementality: the sales that happened because of the tool, not just alongside it.
Why tools all look like winners
Most tools count every sale that came after someone saw or clicked their work. But many of those shoppers were going to buy anyway: they were already on your email list, already searching for your brand, already back for a second order.
When every tool counts the same sale, the credit adds up to more revenue than you actually made.
A holdout fixes this, because the shoppers who were going to buy anyway appear in both groups. Only the extra buying in the reached group counts.
How big the holdout should be
It's a trade-off. A larger holdout gives a clearer answer, sooner, but means fewer shoppers see the tool's work while you test. Smaller audiences need a larger share to get a readable result.
| Connect360 plan | Monthly ad budget | Holdout |
|---|---|---|
| Startup | Under $5K | 10% |
| Growth | $5K–25K | 15% |
| Established | $25K–100K | 20% |
How to read the result
Work it out in three steps. Here's an example with sample figures: a store reaches 100,000 shoppers in a month and holds out 15%.
- Find each group's conversion rate. The 85,000 reached shoppers placed 2,040 orders (2.4%). The 15,000 held-out shoppers placed 300 (2.0%).
- Compare the rates. Lift = (2.4% − 2.0%) ÷ 2.0% = 20%.
- Turn it into orders. The extra 0.4 points × 85,000 shoppers = 340 orders the tool really added. At a $60 average order, that's $20,400.
The tool's own report would likely claim all 2,040 orders, or $122,400. The holdout says it added 340. Both numbers are "true"; only one tells you what the tool is worth.
What did your tool really add?
A simple estimate for learning. Real tests also check the result isn’t down to chance.
Four common mistakes
- Comparing totals. The reached group is bigger, so it will always have more orders. Compare rates.
- Letting the holdout leak. If held-out shoppers still get the same message through another tool or channel, the test understates the lift.
- Stopping early. Results swing in the first days. Pick the length up front, and give new tools time to learn before you score them.
- Moving the goalposts. If the target changes mid-month, nobody can say whether the tool hit it. Lock it for the period.
How Connect360 does it
Every tool on Connect360 is measured against a holdout no tool reaches, scored against a target locked for each 30-day period, and given a 30-day ramp-up before scoring starts. Tools earn a performance fee only when they hit their target.
See how the loop works