Experiment Analysis FAQ

For more information on Experiment Analysis methodology, see our Experiment Analysis Documentation.

Is it ok to end a test early?

In general, Paramark never recommends ending tests prematurely.

There are some exceptions to this rule:

• There’s clear and obvious deterioration in business performance.

• There’s a problem with the test design or set up that is identified during the test period. E.g. there are overlapping test/control geos across two incrementality tests which will make it impossible to analyze lift.

Some examples of false positives or false negatives

• False positive

  • An incrementality test is designed for a 6 week period and the power analysis determines that we can detect a lift of 4%.

  • We observe a lift of 2% in the first two weeks, pre-maturely conclude the test, and call it a win given the positive lift.

  • It’s possible that the 2% lift was within the natural variance of the data set. Running the test for the full duration might have resulted in a no-lift outcome or a true 4% lift outcome but we can’t know because we ended the test prematurely.

• False negative

  • An incrementality test is designed for a 6 week period and the power analysis determines that we can detect a reduction of 4% (in geos where we’re reducing spend).

  • We observe no reduction in the first two weeks, pre-maturely conclude the test, and call it a win given the lack of a reduction.

  • It’s possible that there’s a lagged effect from this type of marketing and 2 weeks wasn’t enough for the reduction to show up in the data set.

Assessing tests in the middle of the time period or based on in-platform metrics is not a statistically valid approach. While it’s tempting to do this, it undermines the purpose of incrementality testing in the first place.

Can I get early learnings before an experiment finishes?

Usually, no. If an experiment is still running, Paramark generally will not provide an official early read or strategic recommendation based on partial results.

There are two reasons:

• The experiment has not yet reached the planned volume and duration. Before that point, any apparent signal may just be normal noise rather than a real business effect.

• Some channels take time to show their full impact. Lagged conversion behavior and post-treatment effects can make a live test look weaker or different than the final readout.

If you’re asking because you want to know whether there are “good signs” so far, the safest answer is that the experiment needs to finish before Paramark can determine whether there was lift, no lift, or an inconclusive result.

The better mid-test question is usually operational, not analytical: is the test still running correctly? During a live experiment, it’s appropriate to confirm that geo targeting, budget levels, exclusions, and holdout setup are behaving as planned. But that is different from drawing conclusions about performance before the experiment is complete.

The main exception is when there is a clear execution problem or an obvious business risk that may require pausing the test. In those cases, talk to your Growth Advisor right away.

Why might conversions or AQLs show up later, especially after the treatment period ends?

That can be completely normal for some channels. A person may see an ad during the treatment window, but not convert until days or weeks later. This is especially common for upper-funnel or consideration-heavy channels such as YouTube, paid social prospecting, TV, or other awareness media.

That is why Paramark sometimes includes a cooldown or post-treatment window in the analysis. The purpose is to capture delayed conversions that were still caused by the treatment, even though they happened after spend stopped.

So if AQLs begin to separate more clearly after the treatment period, the right interpretation is usually not "the test started working late". It is more often that the marketing impact took time to materialize in the target metric. Whether that pattern is expected depends on the channel, the customer journey, and the experiment design.

The important caveat is that delayed conversions are only interpretable within the planned experiment window and any predefined cooldown period. Looking too early, or extending the read indefinitely, can make the result less reliable rather than more reliable.

When can I expect the results of a Google iOS lift test or pulse-up test?

Usually not immediately after launch, and often not immediately when media stops. The right expectation is that Paramark can give a final result only after the planned test window is complete and any required cooldown or delayed-conversion window has had time to mature.

For mobile and platform-based tests, recent days can still be incomplete because downstream conversions may continue to land after exposure. That means an early in-platform read can look noisier, weaker, or simply unfinished compared with the final analysis.

So if you are asking when you will get the result, the safest answer is: after the experiment has finished collecting the planned volume and duration, and after Paramark has allowed enough time for delayed outcomes to settle. The exact timing depends on the channel, the conversion lag, and whether a cooldown period was part of the design.

If you want a mid-test update, the most reliable question is operational rather than analytical: is the test running correctly, is delivery on track, and are the latest days still immature? Paramark generally avoids treating partial platform results as the final answer.

When will I get results from an influencer geo-lift test?

Usually not the moment the campaign stops. Influencer geo-lift tests often need a short cooldown or post-treatment window before the final read is ready, because people may see the content during the test but convert later.

In practice, the right expectation is: Paramark can share a final result after the planned treatment window is complete and after any predefined cooldown period has had time to mature. For many influencer tests, that means the analysis is delivered at the end of the test week or in the first few business days after, depending on conversion lag and data readiness.

So if someone asks, “Do we have results yet?”, the safest answer is usually: not until the planned window is finished and the delayed-conversion period has settled. Before that, Paramark may confirm whether the test is running correctly, but generally avoids calling partial performance the final result.

Why do recent days in a live experiment sometimes look dramatically worse on CAC or cost per signup?

This is often a reporting-timing issue rather than a true deterioration in incremental performance. Spend is usually close to final immediately, but lower-funnel outcomes such as subscriptions, conversions, AQLs, or signups may still be filling in for the most recent days.

When the denominator is still incomplete, efficiency metrics like CAC, CPA, or cost per signup can look artificially high. In extreme cases, a day with spend but fewer than one recorded conversion can appear to have no efficiency at all, even though conversions may still land later.

That means the safest way to read a live chart is to separate settled days from the unsettled tail. If the earlier portion of the window has largely stopped moving, that section is usually the better directional read. The last few days should be treated as provisional until the conversion window has had time to mature.

This does not mean every weak result will improve with time. It means recent-day efficiency should not be interpreted as final while outcomes are still accruing. The right question mid-test is usually not “did CAC spike for real?” but “how much of the latest period is still incomplete?”

Practically, Paramark recommends:

• comparing like-for-like windows rather than over-weighting the newest few days

• checking whether the apparent gap persists on the more settled portion of the test

• avoiding hard conclusions until the planned analysis window and any cooldown period are complete

So if a live experiment suddenly looks much more expensive than its benchmark, the first interpretation should usually be caution: recent conversions may simply not have fully landed yet.

Why can the most recent days in an experiment check-in look worse than the earlier days?

This is often a data-timing issue rather than a true deterioration in performance. In many in-platform readouts, spend, impressions, clicks, and reach are close to final quickly, while downstream outcomes such as subscriptions, signups, conversions, or AQLs continue to fill in over the following days or weeks.

When that happens, efficiency metrics like CAC, cost per signup, or cost per subscription can look artificially high at the end of the window because the numerator has arrived faster than the denominator. A recent three-day tail can therefore look much worse than the more settled portion of the same experiment, even if nothing is actually broken.

The practical rule is:

• Use caution with the unsettled tail. The newest days are usually the least reliable for judging efficiency.

• Compare settled periods to settled periods. If you want a cleaner directional read, compare the earlier fully matured part of the window across test and baseline or control.

• Look for operational issues separately. Delivery problems, geo leakage, targeting mistakes, or tracking breaks should still be investigated, but they are different from normal conversion lag.

So if the right edge of a chart suddenly looks alarming, the first question is usually not "did the experiment fail?" but "are these outcome metrics still settling?" Paramark's final interpretation should rely on the planned analysis window and any defined cooldown period, not on an immature tail.

How do I interpret experiment results?

Our results are provided as a range to help you understand the possibilities. Marketing experiments are rarely perfectly precise.

Practical Significance: We translate statistical findings into business impact - cost per incremental conversion, total incremental revenue, etc.

Actionable Insights: Each analysis includes clear recommendations for budget allocation and channel optimization based on the findings.

How does Paramark select controls?

We typically run dozens of different models for each experiment analysis, varying:

• Control geographies: Different combinations of states, DMAs, or regions

• Model parameters: Various settings within each modeling framework

• Methodologies: MarketMatching, GeoLift

• Data aggregation: Daily vs. 7-day rolling averages

From these many models, we select the 2-4 best-performing controls based on:

1. Pre-period fit quality: How well the model predicts the test regions before the experiment

2. Statistical robustness: Models that avoid overfitting to historical noise

3. Geographic representativeness: Controls that make intuitive business sense

During analysis, we:

• Examine control influence: Identify if any single region has outsized impact

• Test exclusions: Remove potentially problematic controls and rerun models

• Validate consistency: Ensure multiple controls provide similar directional results

We exclude controls when we observe:

• Unexplainable spikes or anomalies in control regions

• Poor model fit during the pre-treatment period

• Geographic or demographic mismatches with test markets

What do Control 1, Control 2, and Control 3 mean in my experiment readout?

Those labels usually refer to different high-quality synthetic control models that Paramark selected for the final readout. They are not separate experiments, and they are not different versions of your treatment. They are alternative counterfactuals — different statistically valid estimates of what would likely have happened in the test markets if the treatment had not run.

Each control model may use a different combination of control geographies, model settings, or methodology. Paramark compares many candidates, then surfaces a small number of the strongest and most interpretable ones.

Seeing multiple controls is intentional. It helps show whether the conclusion is stable across reasonable modeling choices. If Control 1, 2, and 3 all tell a similar story, confidence in the directional result is stronger. If they diverge meaningfully, the result may still be useful, but it usually means there is more uncertainty in the exact size of lift.

So the practical way to read those lines is: not "which control is correct?" but "do the best controls broadly agree on the outcome?" Paramark uses that agreement, along with model quality, to form the final conclusion and confidence level.

Why does Paramark use multiple control models?

Managing Uncertainty

Marketing measurement is inherently probabilistic. Our ensemble approach acknowledges this uncertainty while providing actionable insights within confidence bounds.

• Transparency: Shows the range of plausible effects rather than a single point estimate

• Robustness: Less susceptible to anomalies in any single control

Practical Application

In your experiment readouts, you'll see:

• Multiple control lines: Typically 2-4 synthetic controls shown

• Confidence ranges: Based on agreement between controls

• Sensitivity analysis: How results change when excluding specific controls

This methodology ensures that your experiment results are both statistically sound and practically actionable for marketing decision-making.

Why do AQLs or conversions sometimes show up with a delay during or after a test?

Sometimes, yes. Especially for channels like YouTube, prospecting social, TV, or other awareness-oriented media, people may see the marketing now but convert days or weeks later.

That means the measured lift can emerge gradually rather than immediately. A late-emerging signal does not automatically mean the analysis is wrong — it may reflect the real response window of the channel.

That is also why Paramark may include or closely watch a cooldown or post-treatment period. The goal is to capture conversions that were influenced during the test but happened after the media exposure.

The practical interpretation is:

• Fast-response channels often show most of their effect quickly.

• Longer-lag channels can show more of their effect later, so confidence in efficiency metrics may improve as the cooldown period completes.

This is why Paramark generally avoids making strong mid-test conclusions: delayed response can make an experiment look weaker early than it looks in the final read.

What is the difference between Control 1, Control 2, and Control 3 in an experiment readout?

Control 1, Control 2, and Control 3 are usually different high-quality synthetic control models that Paramark selected from a larger set of candidates.

They are not three separate experiments. They are three different ways of estimating what would likely have happened in the test geographies if the treatment had not run.

Those models can differ because they use different:

• combinations of control geographies

• model parameters

• methodologies or specifications

• data smoothing or aggregation choices

Paramark surfaces multiple controls so customers can see whether the result is directionally consistent across more than one reasonable model. When those controls tell a similar story, confidence is higher. When they diverge, confidence is lower and the result may be more sensitive to modeling assumptions.

In short: the gap between the observed line and each control line shows the possible lift, while the agreement across the controls helps indicate how robust that conclusion is.

Why didn’t I get good results in an experiment?

Experiment design is theoretical. It simulates the potential outcomes of the proposed test design, and we pick the best possible option. Despite this, not all experiments show results, but this doesn’t mean they aren’t successful. The goal of expert experiment is to learn something. It tests a hypothesis, and we frequently are able to get a signal on whether that hypothesis was correct or incorrect.

How are results incorporated back into MMM?

If Paramark receives experiment results that are sufficiently credible and materially informative relative to the current model, we can use them in the next MMM refresh as a calibration input rather than treating them as a separate readout that lives forever on its own.

In practice, that usually means the experiment result becomes an informative prior for the tested channel in the next model run. This does not force the refreshed MMM to exactly match the experiment result. Instead, it guides the model toward the causal evidence while the model still fits against the full historical data set.

A few practical conditions matter before doing this:

• The result needs to be credible enough to use. A noisy, underpowered, or inconclusive test should not be treated as a precise anchor. A medium or high confidence result can be utilized for calibrating priors.

• The tested setup should still match the current measurement question. If the experiment only covered a narrow slice of the channel, a very different geo mix, or an older campaign structure, it may be better used as directional context than as a hard calibration input.

• The result should be meaningfully relevant to the latest MMM. Paramark looks at whether the experimental evidence is materially different from, or helpful in refining, the current model view.

When those conditions are met, the benefit is not limited to the tested channel. Because all channels compete for credit inside an MMM, anchoring one channel with causal evidence can also improve the reliability of the rest of the model.

A simple way to think about it:

• MMM without experiment input: best estimate from historical patterns alone

• Experiment result: direct causal evidence for a specific channel, geo, and time period

• Refreshed MMM: the model’s updated best-fit view after combining that causal evidence with the broader historical record

So when someone asks whether experiment findings can be incorporated into the model, the answer is yes — but usually through calibration, not by forcing the MMM to mechanically copy the test result one-for-one.

What does that mean in practice for a business with multiple completed tests?

If a team has run several credible tests across markets, channels, or periods — for example, separate digital experiments in markets like the UAE and KSA — Paramark does not simply average those outcomes together and overwrite the model. Instead, we review each result for quality, scope, and comparability, then decide whether it should act as a directional input, a stronger prior for a specific channel, or just supporting context for interpretation.

A few practical questions help determine whether those findings can be used in the next MMM refresh:

• Was the test credible? The experiment should be sufficiently powered, cleanly executed, and not obviously inconclusive.

• Does the tested setup match today’s channel definition? If the experiment covered a very specific tactic, audience, or market, we may use it more narrowly rather than treating it as a universal truth for the whole channel.

• Is the result materially informative relative to the current MMM? If the current model and the experiment already tell a similar story, the test may mainly increase confidence. If they differ, the test may justify stronger calibration in the next refresh.

• Can the findings be translated into the model’s channel structure? The closer the experiment maps to the way spend is represented in MMM, the easier it is to use it directly as a calibration input.

So if you have a body of incrementality learnings from multiple digital tests, the right next step is usually to bring those readouts into the next MMM refresh discussion. Paramark can then evaluate which findings are strong enough and specific enough to influence priors, which ones should remain directional context, and whether any channel definitions or modeling windows should be adjusted before integration.

How should I interpret a CTV holdout experiment?

A CTV holdout experiment is still an incrementality test, so the goal is the same as any other experiment: estimate what would likely have happened in the treatment geographies if the CTV campaign had not run, then compare that counterfactual to what actually happened.

The practical difference is that CTV often behaves like an upper-funnel or consideration-heavy channel. That means the immediate in-platform performance metrics may not tell the full story, and the business outcome can emerge with a delay.

When reading a CTV holdout result, focus on these questions:

• Did the treatment geographies separate from the controls? That directional gap is the core evidence of lift or no lift.

• Do the selected control models broadly agree? If multiple controls tell a similar story, confidence is stronger. If they diverge, the result may be more sensitive to modeling assumptions.

• Was there enough time for the channel to work? CTV can produce delayed conversions, so the planned analysis window and any cooldown period matter a lot.

• Are you judging the test on business outcomes rather than platform-reported engagement? Reach, video completion, or site visits can be useful diagnostics, but they are not the experiment result. The real question is whether the target outcome moved incrementally.

So if someone asks, "What did the CTV holdout experiment show?" the best answer is usually not just a platform summary. It should describe whether Paramark found evidence of lift, no lift, or an inconclusive result, how confident that conclusion is, and whether any delayed-response window was part of the final readout.

How should I interpret a CTV holdout experiment result?

A CTV holdout experiment is usually trying to answer one question: did markets, audiences, or households exposed to the CTV campaign perform differently than a comparable holdout group that did not receive that exposure?

In practice, Paramark will usually classify the outcome as lift, no lift, or inconclusive based on how the observed results compare with the expected counterfactual from the control or synthetic control models.

• Lift means the exposed group outperformed the holdout by enough to support the conclusion that CTV drove incremental impact.

• No lift means the exposed group did not clearly outperform the holdout, so the test did not show incremental impact at the measured spend and design.

• Inconclusive means the comparison was too noisy or unstable to make a strong directional call.

CTV tests often require extra care in interpretation because response can lag exposure. Someone may see a CTV ad during the treatment window but convert later, so final readouts may include a cooldown or post-treatment window before the analysis is considered complete.

That means the safest question is not only “did the line move during the campaign?” but also “did the final analysis, after the planned window and cooldown, show incremental lift versus the holdout?”

If you are reviewing a completed readout, the most useful pieces to focus on are the overall lift classification, the confidence level, the size of the estimated impact, and whether multiple controls tell a similar story.

How should I interpret a null result? Could it just be a power issue?

Sometimes a null result really does mean the test found no meaningful lift at the spend level, duration, and design that were actually run. But sometimes it means the experiment was underpowered — in other words, the test may not have had enough signal to reliably detect the size of effect the business cared about.

The key distinction is this: a null result is not automatically the same thing as proof of zero impact. It is evidence that Paramark did not detect clear incremental lift under the observed conditions. The next step is to ask whether the design was capable of detecting a business-relevant effect in the first place.

A few practical questions help make that call:

• Was the test powered to detect the effect size that mattered? If the minimum detectable effect was larger than the lift you would have considered meaningful, then a no-lift outcome may be hard to interpret as a firm "no."

• Was the treatment difference large enough? If spend separation between test and control was smaller than intended, the experiment may have been too weak to generate a detectable signal.

• Did the test run for the planned duration? Ending early, unstable delivery, or insufficient volume can all reduce power and make real effects harder to see.

• Was the KPI slow to respond? Upper-funnel channels and longer-consideration journeys may need a cooldown or post-treatment window before the full effect is observable.

So the practical interpretation is usually one of three things:

• Confident no lift: the test was well-powered, cleanly executed, and still showed no meaningful incremental effect.

• Inconclusive because of power: the hypothesis may still be viable, but the budget, duration, or treatment contrast were too small to support a confident read.

• No lift with a design lesson: the result is directionally useful, but the follow-up decision is to rerun with a stronger design or a faster, more sensitive KPI.

If a team is asking, "Are we confident there was no lift, or was the test just underpowered?" the best answer is usually not a blanket yes or no. It is: look at the original power assumptions, the realized treatment difference, the final duration, and whether delayed outcomes had time to settle. Those factors determine whether the null result should be treated as a business decision or as a signal to redesign and retest.

How do I estimate CAC from an experiment result?

Start by separating two questions that often get blended together: cost per incremental outcome from the experiment versus true CAC.

If you only have experiment spend and estimated incremental conversions, the cleanest calculation is usually incremental CPA: incremental spend divided by incremental conversions. For example, if the treatment added $100,000 in media spend and Paramark estimates 250 incremental checkouts, the implied incremental cost per checkout is $400.

That is often what teams mean in practice when they ask, "what CAC does this imply?" But strictly speaking, CAC usually means the all-in cost to acquire a customer, not just media spend. If you want true CAC, you would usually add any acquisition costs your business includes in CAC such as agency fees, creative, platform costs, or sales support, then divide by the incremental customers attributable to the test.

There are a few practical cautions when using an experiment result this way:

• Use the right denominator. If the experiment result is in incremental checkouts, signups, or leads, that gives you cost per incremental checkout, signup, or lead unless you can translate that outcome cleanly to customers.

• Respect the confidence range. If the lift estimate is a range rather than a single point, the implied cost range can be wide. Lower lift means worse implied CAC; higher lift means better implied CAC.

• Match the spend window to the measured outcome window. If the analysis includes a cooldown or delayed-conversion period, keep that in mind before comparing the result to a platform-reported CAC from a shorter window.

• Do not over-compare to last-touch CAC. A test-based incremental cost metric and a platform attribution CAC are answering different questions, so they often will not match.

So if someone asks for the estimate from a completed experiment, the practical response is usually: take the incremental spend increase and divide it by the estimated incremental outcomes, then be explicit about whether you are reporting incremental CPA or true all-in CAC.

Want to learn more about experimentation? Check out other Paramark experimentation articles here