A holdout test works by deliberately keeping your ad away from a slice of your audience, then comparing what that group does against the group that actually saw the ad. If you're trying to figure out how to set up holdout tests for ads, the short version is: pick a holdout structure that fits your scale (geo, audience percentage, or platform), split the groups cleanly, run long enough to cover a real buying cycle, and compare outcomes against a decision rule you wrote down before launch. The rest of this guide walks through each step with examples you can adapt.
What a Holdout Test Is, and Why It's Different From an A/B Test
An A/B test compares two or more ad variants that are both running — version A against version B. A holdout test compares an ad against nothing. One group of people, accounts, or regions sees the campaign; a matched group sees no campaign at all (or sees unrelated content). The gap between the two groups' outcomes is your estimate of incrementality: the amount of results the ad actually caused, on top of what would have happened anyway from organic demand, brand recognition, or repeat buying behavior.
This matters because engagement metrics like click-through rate or video completion can look strong while the campaign adds very little beyond what organic traffic would have delivered on its own. A holdout test is the cleanest way to answer a harder, more useful question: if I turned this campaign off entirely, would revenue actually drop, and by how much.
How to Set Up a Holdout Test for Ads, Step by Step
Step 1: Define the question you're actually testing
Before touching a campaign setting, write one sentence: "Does running [campaign/channel/creative] produce more [purchases/sign-ups/revenue] than not running it, over [time window]?" Everything downstream — structure, sample size, duration, metric — depends on this sentence being specific. A vague goal like "see if ads work" leads to a vague test that nobody trusts later.
Step 2: Choose a holdout structure that matches your scale
There are a few practical ways to carve out a holdout group, and the right one depends on how much reach and budget you have:
- Geo holdout — pick comparable regions, cities, or zip codes; run ads in some and pause them entirely in others. Works well for brands with enough geographic spread to find matched pairs (similar population, similar past sales).
- Audience-percentage holdout — most ad platforms let you exclude a random percentage of your eligible audience from seeing a given campaign. That excluded slice becomes your holdout. This is the most accessible option for single-region or online-only brands.
- Platform holdout — pause an entire platform or channel (for example, stop all paid social for a defined period) while keeping everything else constant, then compare total sales or sign-ups to the same window a cycle earlier. Blunter, but useful when you want to know if paid social as a whole is pulling its weight.
- Ghost ad / PSA holdout — instead of showing nothing, the holdout group sees a neutral placeholder ad (a public-service message or unrelated brand-safe creative) in the same ad slot. This isolates the effect of your specific creative from the effect of simply occupying that ad placement.
Step 3: Size the groups and set a duration before launch
Two decisions go wrong most often here: holdout groups that are too small to detect anything, and tests that end too soon to capture the full buying cycle. As a starting point to test, don't shrink the holdout below a slice that still gives you a believable comparison group — a token 1-2% exclusion on a small account rarely produces a readable signal. For duration, plan to run at least one full purchase cycle for your product (for a low-consideration item that might be one to two weeks; for anything considered or seasonal, plan longer), and resist the urge to call the test early just because the first few days look directionally clear.
Step 4: Build the measurement plan before you launch
Decide, in writing, which metric is the primary read (revenue, orders, sign-ups) and which decision rule ends the test (for example: "if the exposed group's purchase rate is meaningfully higher than the holdout group's for two consecutive measurement windows, we scale the campaign"). Also decide how you'll handle people who move between groups — someone in the holdout region who travels, or an account that gets reassigned mid-test. Treat any such leakage as a reason to extend the test or shrink the sample you trust, not as something to quietly ignore.
Step 5: Launch cleanly and avoid contamination
Contamination is the biggest practical risk. If the holdout group can still see your organic social posts, email campaigns, or retargeting from an unrelated placement, the test is comparing "some marketing" against "less marketing," not "ads" against "no ads." Before launch, check that: the holdout group is excluded from the same campaign across every placement, other marketing channels aren't quietly re-targeting the holdout audience, and the exposed group isn't also appearing in a different campaign you're running at the same time.
Step 6: Read the results against the rule you already wrote
At the end of the window, compare the two groups on the primary metric only, using the decision rule from Step 4. If the lift is unclear, that's a valid result too — it tells you the campaign isn't adding enough on top of organic demand to justify the spend at its current structure, which is worth knowing even though it's not the answer you hoped for.
Comparing the Main Holdout Structures
| Structure | Best for | How it works | Watch out for |
|---|---|---|---|
| Geo holdout | Multi-region brands with enough stores or regions to find matched pairs | Run the campaign in selected regions, pause it entirely in comparable regions | Finding genuinely comparable regions; local events or weather skewing one side |
| Audience-percentage holdout | Single-region or online-only brands using platform targeting | Platform randomly excludes a set percentage of the eligible audience from the campaign | Excluding too small a percentage to get a readable signal |
| Platform holdout | Brands testing whether a channel matters at all | Pause an entire platform for a period, compare total sales to a prior comparable window | Seasonality and other channels shifting at the same time |
| Ghost ad / PSA holdout | Brands wanting to isolate creative effect from placement effect | Holdout group sees a neutral placeholder ad instead of nothing | Requires platform support for placeholder ads; not available everywhere |
Common Mistakes That Ruin Holdout Tests
- Changing the creative, offer, or budget mid-test, which makes it impossible to say what caused any difference in results.
- Ending the test as soon as the numbers look favorable, instead of sticking to the pre-set duration.
- Letting the holdout group still see retargeting or email promoting the same offer, which quietly erases the comparison.
- Picking a holdout group that isn't actually comparable — a region with a different customer mix, or an audience segment that behaves differently from the exposed group for reasons unrelated to the ad.
- Treating a single holdout test as permanent proof. Markets shift; rerun the test periodically, especially after a major creative refresh or price change.
Holdout Tests vs A/B Tests: When to Use Each
A holdout test tells you whether running ads at all is worth the spend. An A/B test tells you which version of an already-worthwhile ad performs better. They answer different questions and work well in sequence: run a holdout test to confirm the channel or campaign is adding real results, then use an A/B test for video ads to optimize the hook, offer, or call to action within that campaign. Skipping the holdout step means you might spend months optimizing a campaign that was never adding much over organic demand in the first place.
Where Creative Quality Fits Into a Holdout Test
A holdout test can tell you a campaign isn't adding much, but it can't tell you why on its own. Before writing off a channel, check whether the creative itself is pulling its weight — a dull or generic ad can suppress incrementality just as much as weak targeting. It helps to look at supporting signals like video completion rate and click-through rate for the exposed group during the test window; if those are weak too, the fix might be a creative refresh rather than pulling the plug on the channel. If budget is the constraint on running a proper holdout test at all, start by reading how to size a creative testing budget for small businesses so the holdout doesn't eat resources you need elsewhere.
How FrameNotion Fits Into a Holdout Testing Program
Holdout tests work best when the exposed group is seeing an ad genuinely worth testing, not a rushed placeholder. FrameNotion turns a product link or an uploaded poster into a finished 30-second vertical ad — with hook, problem, benefit, proof, and call to action — in about 10-20 minutes, which makes it practical to keep the exposed group's creative fresh across a multi-week holdout window instead of running the same tired ad the whole time. Every ad ships with captions and a beat-matched soundtrack so it holds up whether the viewer has sound on or off, and texts and colors can be adjusted afterward without starting a new ad from scratch. Browse example ads to see the range, and remember that the first ad is free to try before committing to a plan.
FAQ
Frequently asked questions
How long should a holdout test for ads run?+
Plan for at least one full purchase cycle for your product. A low-consideration item might need one to two weeks; a considered purchase or a seasonal product needs longer. Ending the test early because early numbers look favorable is one of the most common ways holdout tests give misleading answers.
How big should the holdout group be?+
There's no universal number, but as a starting point to test, avoid shrinking the holdout to a token slice like 1-2% of your audience on a small account — it rarely produces a readable signal. The right size depends on your total reach and how much variation your sales already show week to week; the smaller your account, the larger the holdout percentage usually needs to be to get a trustworthy comparison.
Can small or single-region businesses run holdout tests?+
Yes. Geo holdouts require multiple comparable regions, but an audience-percentage holdout works for single-region or online-only brands because it relies on platform targeting to exclude a portion of the eligible audience rather than splitting by location.
What's a ghost ad or PSA holdout, and when is it worth using?+
Instead of showing the holdout group nothing, they see a neutral placeholder or public-service ad in the same slot. This isolates the effect of your specific creative from the effect of simply occupying that ad placement, which is useful when you suspect the placement itself (not the message) is driving results.
Do I need a holdout test if I'm already running A/B tests?+
A/B tests tell you which creative variant performs better, but they assume the campaign is worth running in the first place. A holdout test answers that underlying question. Running both — holdout first to confirm incrementality, then A/B testing to optimize within the campaign — gives a more complete picture than either alone.