Classic Split Testing Methodology
All A/B tests involve pitting two or more versions of a page against each other.
Let’s say you have a Control and one Variant. In a typical A/B test, your traffic will be split evenly until you turn off the test. If the Control is performing with an 80% conversion rate, but the Variant only has a 20% conversion rate, 50% of your traffic will still be sent to the variant performing poorly.
It's important to clarify that a 50/50 test doesn't necessarily mean that exactly 50 people will see one version and 50 will see the other. Instead, a simple random sampling algorithm allocates visitors to each test version. Here's a breakdown of how this process typically works:
Random Sampling: When a visitor arrives at a webpage where an A/B test is being conducted, a random sampling algorithm determines which version of the test they will be exposed to. This randomness ensures that each visitor has an equal chance of being assigned to either test version.
Equal Probability: In a 50/50 test, both versions of the test have an equal probability (50%) of being shown to any given visitor. This means that over a large number of visitors, approximately half will be exposed to version A and half to version B.
Statistical Significance: The goal of A/B testing is to determine whether one version of a webpage performs better than another in terms of predefined metrics such as conversion rate or click-through rate. To draw valid conclusions from the test results, ensure that a sufficient number of visitors are included and that statistical significance is achieved. Here is a recommended calculator for determining this value.
In summary, while the term "50/50 test" implies an equal split between two versions, a random sampling algorithm determines the allocation of visitors. This ensures fairness and statistical validity when evaluating the performance of different web page versions.
Still, have doubts about the success of the Classic Split A/B Testing Methodology?
Ask The Saturday Solopreneur Justin Welsh! After testing his theories with our AB Tester, Justin increased his conversion rate by 39.7%. The results don't lie.