A/B test significance calculator

The conversion rate and the confidence level are automatically calculated with the result

A - Original:
Visitors
Conversions
Conversion rate
Confidence level
B - Variant 1:
Visitors
Conversions
Conversion rate
Confidence level
Icon

Usage information

Icon

Number of test variants

Use the "add variant" and "remove variant" buttons to determine the number of test variants.

Icon

Number of visitors

Please enter the number of visitors of the original page and the number of visitors of the variants of the page in the fields of the column visitors.

Icon

Number of conversions

In the fields of the column conversions, please enter the respective number of conversions (not the conversion rate).

Significance and knowledge

Is the result of my A/B test significant?

If you test different versions of a page against each other on your company's website in an A/B test, the best-case result is that the conversion rate of the new variant is higher than that of the original page. To rule out the possibility that this result just happened by chance, you should perform a significance test. To do this, simply enter the measured values in the corresponding fields in the calculator.

The mathematics behind it

The converlytics A/B test significance calculator is based on a chi-square test. The chi-square test checks whether the relative conversion frequency of visitors who are shown the variant of the page differs significantly from the relative conversion frequency of visitors who are shown the original version of the page. The test size for two versions of the web page is as follows

Formula CHI

and is calculated from the following quantities:

o: visit the original page
v: visits to the variant
co: conversions of the original page
cv: conversions of the variant
n: total number of visits
nf: total number of visits without conversion
nc: total number of visits with conversion

The following applies:

n = o + v
nf = n - ( co + cv )
nc = co + cv

The degrees of freedom result from the number of test cases. Here k=2, since there are the two possibilities conversion and no conversion and m=2, since there are two versions of the website. If there are several versions, the number of degrees of freedom changes. a is the significance level corresponding to the confidence level and is calculated from confidence level=1-alpha. The critical values of the chi² distribution are for example

at confidence level 95%
( alpha = 0.05 ) : Chi²1-alpha (( k-1 ) * ( m - 1 )) = Chi² 0.95 (1) = 3.84

at confidence level 99%
( alpha = 0.01 ) : Chi²1-alpha (( k-1 ) * ( m - 1 )) = Chi² 0.99 (1) = 6.63

If chi² calculated above is greater than the critical value at confidence level 95%, it can be assumed that the difference in conversion rate between the two variants of the website did not occur by chance. Do you want to know more about this? Then get in touch with us.

references