A/B test duration calculator

How long does the A/B test need to run?

For visitors to the test page per day, please enter the average number of visitors per day to the test version of the page. (Visitors who are shown the original page are not counted).

As conversion rate please enter the current ratio of the number of conversions via the original page to the number of visitors on the original page.

The expected change in conversion rate is the percentage change expected from the conversion rate of the original page to the conversion rate of the test variant. If you expect a significant change, you can enter 25%, if you expect a less significant change, you can enter 15%.

Duration calculator

The converlytics A/B test duration calculator calculates the A/B test duration required for a statistically significant result.




















Icon

Elimination of the disturbing influencing variables

The A/B test or multivariate test is designed to eliminate different influences on different test groups. The test groups are shown different variations of the website at the same time so that all external factors are equal.

An example

As an example, we will look at the online ticket sales of a large cinema. The design of the cinema website is very colorful and full of different graphic elements in the original (A) and very simple and tidy in the test variant (B). Now, it may happen that on weekdays, children and teenagers are more likely to buy movie tickets, which are more attracted by the colorful design. Thus, on weekdays, the original site would perform better than the test variant. If, on weekends, many more adults buy cinema tickets who are attracted by the simple design, this would change and the test version would perform better. If, in addition, more tickets were sold in absolute numbers at the weekend than during the rest of the week, the decision must be made in favour of the test version.

However, if the A/B test had only been carried out on five weekdays, the cinema operators would have opted for the colourful original page without noticing that the weekend sales figures would have reversed the ratio.

Special features of the A/B test!

Nevertheless, even if the necessary number of visitors may have been reached earlier, an A/B test or multivariate test should have a certain minimum duration. The reason lies, among other things, in the seasonality and in the fact that a positive test result is also only significant for the tested period.

Recommended minimum test duration

Thus, in order to eliminate at least the seasonally fluctuating influences of weekdays, a test should run for at least 7 days. In order to run the weekly cycle twice and thus make the results even more stable, converlytics recommends that an A/B test or a multivariate test be scheduled for at least 14 days.

By the way, an interesting approach would be to consider showing a different version of the website during the week than on the weekend. Always provided that this does not cause confusion among customers.

The mathematics behind it

Error probability

An A/B test examines whether the conversion rate of the test variant and the conversion rate of the original page differ from each other. As a result, there are the two possibilities that the test either shows a difference or that it shows no difference. The test can thus be "wrong" in two different ways. On the one hand, the test can detect a difference even though there is no difference in reality (probability alpha) and on the other hand, the test can detect no difference even though there is a difference in reality (probability beta). Justification: A/B statistical testing relies on probabilities. This means that no result is 100% certain.

However, the test duration can be calculated sufficiently large, so that the probability of making a mistake is relatively small. Common values are 0.2 to 0.025 for beta and 0.1 to 0.01 for alpha.

Calculation

The confidence level 1-alpha indicates the probability with which it can be assumed that a difference between the conversion rates is not random. The test strength (power) 1-beta indicates the probability that an undetected difference between the conversion rates does not exist in reality. A power of 1-beta=95% to the 99% confidence level and a power of 1-beta=80% to the 95% confidence level are fixed in the calculator. The calculation is based on the following formula for case number planning, where z denotes the corresponding quantile of the standard normal distribution. co is the current conversion rate of the original site and i is the expected improvement of the conversion rate by the test site.

formula case number

The number of necessary visitors n must then be divided by the number of visitors that your website has daily. This gives you the duration in days that you should schedule for an A/B test on your company's website. Do you want to know more? Then get in touch with us.

references