We (almost) never recommend ending an experiment early. But if, for some other constraint, an experiment ends early, here's what to expect.

The short version

The lift required shown in your experiment design was calculated for a specific test duration. If the test ran shorter, the actual lift required to have a high confidence result is higher. Running the test for fewer weeks means fewer datapoints, which raises the threshold for what the test can reliably confirm.

Your Growth Advisor will account for this in the analysis and tell you the adjusted lift required for the actual run length.

Why the lift required to have confidence changes with duration

An experiment's ability to detect an effect depends on how many observations it accumulates. Observations come from two sources: the number of geo markets in the test and the number of weeks it runs. More of either means more statistical power and a lower MDE.

What this means for your results

Your results can still be valid even if the test ran short. The key question is whether the effect observed (if any) is large enough to clear the adjusted lift required.

• If the measured lift is above the adjusted lift required: the result is statistically reliable at your original confidence level.

• If the measured lift falls below the adjusted lift required: the test was not powered to confirm a result of that size at the original confidence level. This does not mean no effect exists, only that the test as-run could not confirm one at that threshold.

Your Growth Advisor will interpret your results against the adjusted MDE and note both figures in the writeup so you have the full context.

Common questions

Is the experiment invalid if it ran short?

No. A shortened test is sometimes still interpretable. The analysis adjusts for the actual duration. Results that clear the adjusted MDE are reliable; results below it are uncertain.

The power curve shows 80% power at 4-5%. Is that the MDE for my test?

Only if the test ran for the full designed duration. If it ran shorter, the actual MDE is higher. Your GA will calculate and communicate the adjusted figure.

Can we re-run the test for the remaining weeks to get back to the original MDE?

This is sometimes possible for channels where re-starting a geo test is practical. Discuss with your Growth Advisor. Resuming a test after a gap introduces complications (changes in market conditions, seasonality) that may affect the analysis. In most cases, analyzing what ran is the cleaner path.

Want to learn more about experimentation? Check out other Paramark experimentation articles here

Related Articles:

📄 Experiment designs and how to read them

📄 Can I speed up an experiment by spending more? What to know about budget and timeline

📄 Experiment Analysis FAQ