Overview

When you run an incrementality test, there's often a desire to measuring many outcomes. Marketers look at a long list of KPIs — conversions, revenue, ROAS, CTR, AOV, retention, engagement, and so on. The “vast search problem” is what happens when you scan all of those metrics looking for something that moved. The more metrics you check, the easier it becomes to “discover” an effect that isn’t real.

This is a statistics issue, but it has very real marketing consequences. Here’s the intuitive version. When you run an incrementality test, you’re trying to separate real causal impact from random noise. And marketing data is very noisy — impressions fluctuate, baseline demand shifts, and metrics bounce around even with no marketing running at all.

So we use a simple framework:

1. Start with a null hypothesis: “The campaign had no effect on this metric.”

2. Measure the observed difference between test and control.

3. Ask: “Is this difference bigger than what randomness alone could plausibly generate?”

You need some statistical yardstick to answer that question. That yardstick is the p-value.

1. What a p-value actually represents

Every KPI you test starts with the same null hypothesis: “The campaign had no impact on this metric.”

A p-value tells you the probability you’d see a result that strong or stronger purely by chance if the campaign truly had no effect.

Example: With p-value = 0.05, you accept a 5% risk of a false positive for that single metric. Paramark, like other incrementality testing platforms, uses a looser threshold (p-value = 0.20) because marketing/media data are noisy and samples are small. This increases the risk of false-positives risk even further.

2. When you test a single metric

One metric = one shot at accidentally fooling yourself.

• At p-value = 0.05 → 5% chance of a false positive

• At p-value = 0.20 → 20% chance of a false positive

That’s manageable — if you only test one thing.

3. When you test many metrics (the vast search problem)

Every additional metric is another opportunity for randomness to look like a win.

If each metric has a 20% false-positive risk, then:

• 1 metric → 20% chance of a false positive

• 2 metrics → 36%

• 3 metrics → 49%

• 4 metrics → 59%

• 5 metrics → 67%

The math here is simple. Probability of at least one false positive = 1 – (1 – α)ⁿ

Where:

• α = false-positive rate per metric

• n = number of metrics tested

This compounding effect is the heart of the vast search problem.

4. Why marketers should care

If you check enough KPIs, something will look significant just by luck. That can lead to:

• Declaring lift where none exists

• Over-crediting a campaign based on a noisy KPI

• Optimizing future spend around random fluctuations

• Telling a “success story” that collapses on repeat testing

It’s the experimental equivalent of throwing darts blindfolded until you hit a bullseye — then telling everyone you’re a sharpshooter.

5. How to avoid being misled

There are a few practical approaches:

1. Define primary metrics up front. Commit to the KPIs you care about before looking at results.

2. Limit secondary metrics. Use them for directional learning, not “proof.”

3. Re-run suspicious findings. True effects replicate. False positives don’t.

Bottom line

Running an incrementality test is easy. Interpreting it honestly is harder — because the more metrics you inspect, the more the math works against you. The vast search problem is simply the statistical reminder that if you go looking for treasure in enough places, you’ll eventually find “gold” that isn’t real.

This isn’t a reason to fear experimentation. It’s a reason to discipline it.

Want to learn more about experimentation? Check out other Paramark experimentation articles here