A/B Testing: How to Run Tests That Teach You
Learn how to design and interpret A/B testing experiments, understand p-values, statistical power, and calculate required sample sizes.
Stock photo for illustration only, not from the actual event
- Statistical significance does not mean business significance
- P-values do not represent the probability of your change working
- Sample size calculations require baseline data and effect sizes
- Low traffic sites may find A/B testing impractical
Hypothesis testing, p-values, confidence intervals, statistical power, and the traps that fool even experienced teams are critical concepts that developers and marketers must understand to make experiments truly valuable.
In the 1970s, Pepsi ran a marketing campaign using blind taste tests, showing more people preferred Pepsi over Coke. This prompted Coke to panic, reformulate, and launch 'New Coke' in 1985, creating one of the most infamous product launches in history.
The problem was the test itself. A single sip from a small cup measures immediate sweetness preference, not long-term consumption habits. While the blind test achieved statistical significance, it lacked practical significance.
Stock photo for illustration only, not from the actual event
Every controlled experiment follows the same core skeleton, whether testing a medical drug, website headline, or pricing page. It begins with a hypothesis, random splitting of subjects into control and treatment groups, and measuring outcomes against random chance.
"statistical significance does not mean business significance"
Dev.to
The statistical step involves setting up a null hypothesis (H0) and an alternative hypothesis (H1). The answers are driven by the p-value, which calculates the probability of observing a measured difference assuming the null hypothesis is true.
Properly interpreting p-values is crucial. A p-value of 0.03 does not mean there is a 97% chance your change works; it means there is a 3% chance you would see such a difference purely by chance even if your change did nothing.
When drawing conclusions, teams must watch out for Type I errors controlled by significance levels (alpha) and Type II errors related to statistical power, with 80% power and 5% significance acting as a common standard minimum.
The required sample size depends on four key numbers. For a 5% baseline and a 10% relative effect size, roughly 15,700 users per variation are required, scaling up significantly for smaller targeted lifts.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment