How MMM and incrementality testing work together

When Paramark delivers your first MMM results, you'll often hear something like: "The model gives you the direction; experiments confirm it." That framing is accurate, but it undersells what's really happening. These two tools don't just complement each other. They form a measurement system that gets more accurate every time you use it.

This article explains how MMM and incrementality testing work together, why experiments matter even when your model looks healthy, and how running an experiment actually makes your MMM more reliable going forward.

What Each Tool Does

Your MMM and your incrementality tests are asking different questions from different angles.

The MMM looks at your full history of spend and outcomes (typically 18 to 24 months) and asks: "When a channel's spend went up or down, how did the business respond, after accounting for seasonality, growth, and everything else running in parallel?" It gives you a portfolio-wide view of efficiency across all channels at once. It's the strategic map.

An incrementality test asks a more targeted question: "If we change spend on this specific channel in these specific markets during this specific window, does the outcome move?" It gives you a causal answer, not a correlational one. It's the direct measurement.

Neither tool is better than the other. They answer different questions, and the best measurement systems use both.

What kinds of incrementality tests count here?

When this article says incrementality test, it is referring to a family of causal measurement approaches — not just one test format. The right test depends on the channel, the data you can observe, and the business outcome you care about.

• Geo-lift tests are often the default for MMM calibration because they work across many paid channels and can measure downstream business outcomes at the market level.

• Platform conversion lift studies can be useful on channels like Meta, Google, LinkedIn, or TikTok when tag coverage and conversion measurement are strong. They can produce fast, lower-uncertainty directional reads, but they may miss offline or delayed outcomes.

• User- or account-level holdouts are often the better fit for channels like direct mail, email, SMS, or other audience-based programs where the cleanest comparison is between people or accounts that did and did not receive the treatment.

The important principle is not that every channel should use the same experiment format. It is that every major decision should be informed by the cleanest causal method available for that channel. When those results are later used to inform an MMM, Paramark looks at how comparable the methodology and business context are before using the result as a strong calibration input.

Why Experiments Matter Even When Your MMM Looks Good

A common question: if the MMM is already telling you which channels are efficient, why run experiments?

The short answer is that the MMM is working with patterns. When it shows a channel as efficient, it's saying: "historically, when this channel's spend moved, outcomes moved with it." That pattern is usually meaningful. But patterns can reflect things beyond the channel's direct impact, including seasonality, correlated spend across channels, and demand cycles that coincide with when you advertise.

An incrementality test cuts through that. It deliberately changes one thing and measures the result. That kind of direct measurement is the only way to say with confidence: "this channel caused lift, not just coincided with it."

There's also a practical reason related to your budget decisions. Your MMM assigns a confidence level to every channel estimate. When a channel you care about carries medium or low confidence, the model's best estimate comes with wider uncertainty. An experiment narrows that uncertainty. Before you make a significant budget shift (scaling a channel the model says is efficient, or cutting one it says is underperforming), causal validation is what lets you act with conviction rather than hope.

Does Running Experiments Mean the MMM Is Wrong?

No but it is an important discussion.

When your Growth Advisor says experiments will "improve" or "calibrate" the model, it's natural to hear that as "the model was wrong." But that's not what's happening. The MMM is doing the best job possible with the data available to it: historical patterns, spend, outcomes, and seasonality. It produces the most informed estimate it can.

An experiment adds something the model can't generate from historical data alone: direct causal evidence from a controlled test. When that evidence comes in, it doesn't replace the model's work. It anchors one channel's estimate to something we know causally, which makes the whole model more reliable going forward. Think of it less as correction and more as calibration.

A useful analogy: imagine you're using a map to navigate an unfamiliar city. The map is accurate and useful. Now imagine you confirm one landmark by actually standing next to it. That confirmation doesn't mean the map was wrong. It means you now have one reference point you know for certain, which makes everything else on the map easier to trust.

How Experiments Improve the MMM: Simple Terms

When a geo experiment returns a result (say, a measured cost-per-incremental for Facebook ads), that number is fed into the next MMM refresh. The model is then asked to find the best explanation for your historical data given that we know, from direct evidence, that Facebook's efficiency sits around that measured value.

Two things happen as a result. The model's estimate for the tested channel becomes more precise. And because all channels compete for credit in an MMM, anchoring one channel's estimate more firmly makes the estimates of every other channel more reliable too. One well-validated channel improves the whole model.

Over time, as you accumulate experiment results across more channels, the model gets progressively more accurate. Not because the historical patterns changed, but because more of its estimates are anchored to causal evidence rather than inference alone.

How Experiments Improve the MMM: A Closer Look at the Math

For those who want to understand the mechanics more precisely: Paramark's models are Bayesian.

In Bayesian modeling, every parameter has a prior (the distribution of beliefs about a value before seeing data) and a posterior (the updated distribution after the data has been fit). By default, Paramark's channel priors are weakly informative. They allow the historical data to shape the estimates without imposing strong assumptions. This is the right approach when you have no external causal evidence about a channel's true incrementality.

When an incrementality experiment runs and produces a measured result, that result becomes an informative prior for the next model run. The model is now being fit with a constraint: this channel's incremental efficiency should be consistent with the experimental estimate. This narrows the posterior for that channel's parameter. Because the model allocates total credit across all channels simultaneously, tightening one channel's estimate propagates through the model and reduces uncertainty everywhere else too.

The practical outcome: confidence intervals narrow, recommendations become more actionable, and the gap between what the model estimates and what experiments confirm shrinks. More experiments mean more anchored parameters, which means a more reliable model across the board.

The Measurement Flywheel

The right way to think about MMM and experiments together is as a flywheel. Your MMM identifies where your highest-value hypotheses are: which channels look efficient but haven't been causally validated, which look underperforming but deserve a closer look. You design experiments around those hypotheses. The experiments return causal evidence. That evidence is fed into the next model refresh, making it more accurate. A more accurate model generates sharper hypotheses, which lead to better experiments.

Every turn of the flywheel increases confidence. Early on, some decisions will rest on the model's directional signal with medium confidence. Over time, as experiments accumulate, more decisions rest on causal evidence. That's not a limitation of the early model. It's the system working as designed.

How to Talk About This Internally

If you need to explain to leadership why your team is running experiments alongside the MMM, here are a few ways to frame it.

"The MMM tells us which channels to pay attention to. The experiments give us the causal proof to act on them at scale. We're using them together so our big budget decisions are backed by both pattern recognition and direct evidence."

"Running experiments doesn't mean the model is wrong. It means we're building toward a measurement system where every major channel has been directly validated. That makes all of our decisions more defensible."

"The model gets better every time we run an experiment. Each test result improves the model's estimates for the tested channel and makes the rest of the model more reliable. It's a compounding investment."

What the powered minimum CPIO means — and what it does not mean for your next MMM

Another common question is whether the minimum cost per incremental outcome from an experiment design is the number you should expect your MMM to move to after the test. Not exactly.

The powered minimum CPIO is a planning threshold. It answers: "What level of efficiency would this experiment need to be able to detect with confidence, given the proposed spend, duration, and background noise?" It is not a forecast, and it is not a promise that the next MMM refresh will land at that exact value.

That distinction matters because experiment design happens before the result exists. At that stage, Paramark is not saying, "we expect the channel to deliver this exact CPIO." We are saying, "if the true effect is at least this strong, this test has a reasonable chance of detecting it."

Once the experiment is complete, the number that matters for MMM calibration is the measured experimental result — typically expressed as lift, incremental outcomes, or implied cost per incremental outcome — along with its uncertainty. That observed result is what informs the next MMM refresh, not the original power threshold on its own.

Even then, you should not expect the next MMM to mechanically snap to the experiment's exact CPIO. The experiment becomes an informative prior, which means it guides the model toward the causal evidence while the model still fits against the full historical data set. The refreshed MMM may move meaningfully toward the experimental result, but it can still land above or below it depending on the rest of the evidence in the data.

A simple way to think about it:

• Powered minimum CPIO: the weakest effect size the test was designed to reliably detect

• Observed experimental CPIO: what the test actually measured

• Refreshed MMM estimate: the model's updated best-fit view after combining the experimental evidence with the broader historical record

So if you are reviewing an experiment plan, treat the powered minimum CPIO as a design sensitivity threshold. If you are reviewing a completed experiment, treat the observed result as the key causal input. And if you are reviewing the next MMM refresh, expect it to become better anchored by that evidence — not necessarily identical to it.

Building a strong measurement system takes time, and it's normal to start with directional confidence and build toward causal validation. Your Growth Advisor can help you identify which experiments to run first, how to interpret results as they come in, and how each result is incorporated into the model going forward. If you have questions about how this applies to your specific measurement setup, reach out and they'll walk you through it.

Why wouldn't Paramark apply a hard prior from a past channel experiment to the current MMM?

Because a past experiment is only a clean prior for the current model if it is still measuring meaningfully the same decision context.

In practice, Paramark looks at three things before treating an older experiment result as a strong calibration input:

• Was the experiment powered enough to produce a useful signal? If the original test was underpowered, a null or noisy result should not be treated as a precise anchor for the next MMM refresh.

• Is the channel setup still comparable? If the earlier test covered only a subset of formats, placements, audiences, or campaign objectives, it may not represent the channel as it runs today. For example, a LinkedIn test focused on conversation ads and sponsored content may not cleanly transfer to LinkedIn more broadly if the current mix also includes ABM, CTV, or thought-leadership programs.

• Is the business question the same? A past result can inform today's planning, but if the current decision is about a new spend level, a new campaign mix, or a newly separated channel inside the MMM, the safest move is often to use the old result as context rather than as a hard constraint.

That does not mean the old experiment was useless. It still helps frame expectations, highlight what would have been detectable, and inform whether a fresh experiment is worth running. But Paramark is usually careful not to overstate what an older or narrower experiment proved, especially when the channel operating conditions have changed.

A simple rule of thumb: the closer the old test matches today's channel mix, spend level, and measurement question, the more confidently it can be used as a prior. The farther today's setup has drifted, the more likely Paramark is to rely on the result as directional context and recommend a fresh experiment for a decision-quality read.

Will previous incrementality tests automatically be baked into the next version of my MMM?

Not automatically in every case.

A prior experiment can be incorporated into a future model refresh when it is still a good match for the current modeling setup. In practice, that means the earlier test needs to be sufficiently powered, still relevant to the way the channel is running today, and aligned to the decision the refreshed model is trying to answer.

If the new model version is covering a meaningfully different setup — for example a broader channel definition, different campaign mix, new data inputs, or a different business question — Paramark may reference the old experiment as useful context without treating it as a hard calibration input. In those cases, a fresh test is often the cleaner way to inform the next refresh.

So the right expectation is: previous incrementality tests can inform the next MMM, but they are only "baked in" when they still cleanly apply to the current model context.

Will a new MMM automatically include past incrementality tests?

Not always, and that is usually the right call rather than a sign that something was missed.

A past experiment is most useful when it still matches the channel you are modeling today. If the spend level, market mix, campaign objective, audience targeting, placements, or creative strategy have changed meaningfully since the test ran, Paramark may decide not to carry that result forward as a hard prior in the next model version.

In practical terms, that means an earlier incrementality test can show up in one of two ways:

• As a direct calibration input when the old test is still a close match for the current channel setup and produced a strong enough signal to anchor the model.

• As directional context when the test was narrower, older, underpowered, or based on a materially different setup than the one reflected in the new MMM.

So if you hear that a previous test is not "baked into" the newest MMM yet, the right interpretation is usually: Paramark is being careful about whether that old causal evidence still cleanly applies to the current version of the channel.

The next step is not necessarily to force the old result into the model. It is to ask whether the earlier test still matches today's decision context closely enough to be a valid prior, or whether a fresh experiment would give a better anchor for the next refresh.

A helpful way to frame this internally is: past tests are valuable, but they are only directly portable when today's channel setup is still substantially the same as the one that was tested.

Why might Paramark wait for a final CRM incrementality result before recalibrating the MMM?

Because early experiment readouts are useful for direction, but the model should usually be calibrated to the final verified causal result rather than a provisional estimate.

In practice, an early estimate can still change after teams finalize CRM match logic, attribution windows, deduplication, conversion definitions, or late-arriving outcome data. If Paramark anchored the next MMM refresh to a number that later moved, the model would be learning from a temporary signal rather than the cleanest available evidence.

That is why the usual sequence is:

• Use the early result as context for discussion and planning.

• Wait for the final confirmed result when the test depends on CRM or downstream conversion reconciliation.

• Use the verified result to inform calibration of the next MMM refresh.

This does not mean the earlier read was ignored or that the MMM was wrong. It means Paramark is being careful about which version of the causal evidence is strong enough to use as a modeling anchor.

A simple way to explain it internally is: preliminary incrementality results can inform the conversation, but final reconciled results are the safer input for recalibrating the model.

Want to learn more? Check out other Paramark articles here