This article explains what usually happens between deciding to run a holdout test and the day the test launches. It is meant for teams preparing to execute a holdout in-market with Paramark.

If you need background on how experiment designs work, start with Experiment designs and how to read them.

What a holdout test setup conversation is for

Once the design is approved, the next step is usually a working session to make sure the launch can happen cleanly. The goal is not to revisit the theory of incrementality testing. The goal is to translate the design into an executable launch plan.

That setup conversation usually covers:

• the planned launch date

• which markets are in test and control

• which campaigns or ad sets need to change

• what the treatment actually is (for example, pausing spend in holdout markets or increasing spend in test markets)

• who on the customer side will make the platform changes

• what needs to stay stable during the test window

What to have ready before the setup call

The fastest setup calls happen when a few details are already confirmed.

• The experiment design. You should have the final market assignments, recommended duration, and budget or treatment guidance.

• The channel execution plan. Be clear on which platform, campaigns, and geographies will be changed at launch.

• The owner of implementation. Someone should be identified to make the changes in the ad platform on launch day.

• The target launch window. A specific date helps everyone work backward from the launch and flag blockers early.

• Known business events. Major promotions, pricing changes, product launches, or other unusual events should be surfaced before launch because they can affect interpretation.

What Paramark typically does in setup

Paramark's role is to help make sure the experiment is faithful to the approved design.

• confirm the treatment and control structure

• review how the campaign should be configured by market

• call out any risks to keeping non-test markets stable

• pressure-test whether the planned launch still matches the version of the channel the design was built for

• answer questions about duration, cooldown, and what will happen after launch

What the customer team usually owns

The customer team typically owns the in-platform execution.

• implementing the geo targeting changes

• applying the planned holdout or budget treatment

• keeping control markets stable

• avoiding mid-test campaign changes that were not part of the plan

• alerting Paramark if something unexpected changes before or during launch

Questions that are worth resolving on the call

If these questions are answered before the meeting ends, launches tend to go more smoothly:

• Are the final test and control markets locked?

• Is the launch date still realistic?

• Does the campaign structure in the ad platform match the design assumptions?

• Who is making the changes, and when?

• What should remain unchanged during the test?

• Are there any upcoming events that could distort the early test window?

What happens after the setup call

After the setup call, the next steps are usually straightforward:

1. Finalize the launch date.

2. Apply the platform changes that create the holdout or treatment.

3. Launch the test according to the approved plan.

4. Keep the experiment stable for the recommended duration.

5. Allow for any planned cooldown period before final analysis.

When to revisit the design before launch

Sometimes a setup conversation reveals that the design should be refreshed before the test starts. That is usually worth doing if the baseline channel setup changed materially after the design was created.

Examples include:

• a major shift in baseline spend

• a meaningful change in campaign structure or targeting

• a change in how branded and non-branded traffic is routed

• a launch timeline that moved far enough that the original assumptions are stale

If the changes are minor, the design can often still be used with a quick sanity check. If the changes are material, refreshing the design before launch usually leads to a cleaner read.

A simple checklist for launch readiness

• Launch date agreed

• Test and control markets confirmed

• Treatment defined clearly

• Campaign owner identified

• Known risks or business events discussed

• Control markets expected to remain stable

• Paramark and customer teams aligned on next steps

Want more context on experiment planning? See Experiment Design FAQ and Experiment designs and how to read them.