OBJECTIVE
Test two or more
items/objects and identify the one with the best performance.
DESCRIPTION
A/B testing is part
of a broader group of methods used for statistical hypothesis testing in which two
data sets are compared. Having defined a probability threshold (significance
level), we can determine statistically whether to reject the null hypothesis or
not. Usually, the null hypothesis is that there is no significant difference
between the two data sets.
A/B testing is a
randomized experiment with two variants (two-sample hypothesis testing), but we
can also add more samples. The difference from multivariate testing is that in
A/B testing only one element varies, while in the other test different elements
vary and we should test several combinations of elements. These tests are used
in several sectors and for different business issues, but nowadays they are
quite popular in online marketing and website design.
Output of Conversion Rate A/B Testing
Usually the steps to
follow are:
- - Identify the goals: for example “improving the conversion rate of our website”;
- - Generate hypotheses: for example “a bigger BUY button will convert more”;
- - Create variables: in our example the element to be modified is the BUY button and the variation website can be created with a double-size BUY button;
- - Run the experiment:
o
Establish
a sample size: depending on the expected conversion rate, the margin of error that
is acceptable, the confidence level, and the population, the minimum sample
size can be calculated (see the template);
o
The two
versions must be shown to visitors during the same period, and the visitors
must be chosen randomly (we are interested in testing the effect of a larger
button; if we do not choose visitors randomly or show the two versions during
different periods, the results will probably be biased);
-
Analyze the
results:
o
Significance:
depending on the significance level chosen for the test (usually 90%, 95%, or
99%), we can be X% confident that the two versions convert differently;
o
Confidence
intervals: depending on the confidence level chosen, there will be a probable
range of conversion rates (we will be X% confident that the conversion rate
ranges from X to Y);
o
Effect
size: the effect size represents the difference between the two versions.
The proposed template provides
a simple calculator for the necessary sample size and for testing the
significance of conversion rate A/B testing. However, a considerable amount of
information about A/B testing is available online.[1]
The template of chapter 44.
TEST OF PROPORTIONS shows the same test with more statistical
detail as well as the calculation of the mean difference confidence interval,
while the A/B testing template presents confidence intervals for each mean of
the two samples.
In the proposed
example, the data are obtained using a web analytics tool (for example Google
Analytics), but they can come from any experiment that we decide to run.
TEMPLATE
No comments:
Post a Comment