Statistics with Python Specialization (Michigan)
A three-course University of Michigan series teaching statistical visualization, inference, and regression modeling in Python, at beginner level.
A/B testing is a controlled experiment that randomly splits users into two groups: A (the control, which sees the current version) and B (the variant, which sees a change). You compare a pre-chosen metric, such as conversion rate, between the groups. Because assignment is random, a difference that is big enough to rule out chance can be attributed to the change itself. It's the most practical way to show causation in business data.
Dashboards show what happened. A/B tests show what a change caused. That's why companies test everything from button colors to pricing, and why "how would you design an A/B test?" is a standard analyst interview question.
If you roll out a new checkout page and compare this month's conversion to last month's, anything else that changed (season, marketing, competitors) is mixed in with your change. If you compare users who chose a new feature with users who didn't, you get selection bias.
Random assignment removes both problems. The two groups are the same on average in every way, measured or not, except for the change you made. So when the outcome differs by more than chance would explain, the change caused it.
An online store wants to test a simplified one-page checkout (B) against the current three-step checkout (A).
"A one-page checkout will increase purchase conversion rate, because fewer steps mean fewer drop-offs."
Choose these before the test starts. Picking the metric after seeing the results is one of the fastest ways to fool yourself.
Current conversion is 4.0%. The team decides a lift to 4.5% (+0.5 percentage points) would be worth the engineering effort.
The standard sample-size calculation depends on three inputs:
With those inputs, the answer is roughly 25,500 users per group, about 51,000 in total. With 3,000 checkout visitors a day, the test needs around 17 days. The team plans to run it for three full weeks, so every weekday appears the same number of times.
Online sample-size calculators and Python's statsmodels do this calculation for you. What matters is doing it before you start, not stopping when the results look good.
Users are randomly assigned, 50/50, and each user always sees the same version. The team checks that the split is actually close to 50/50. A mismatch (called sample ratio mismatch) signals a bug in the assignment, and the results can't be trusted.
| Group | Users | Purchases | Conversion |
|---|---|---|---|
| A (current) | 25,500 | 1,020 | 4.00% |
| B (one-page) | 25,500 | 1,160 | 4.55% |
Decision: roll out the one-page checkout to everyone.
| Term | Plain meaning |
|---|---|
| Control / variant | Current version / changed version |
| Null hypothesis | "The change has no effect." The test asks whether the data is surprising under this assumption |
| p-value | How surprising the observed difference would be if there were no real effect. Not the probability that B is better |
| Significance level (α) | The threshold for "surprising enough", usually 5% |
| Statistical power | The chance of detecting a real effect of the chosen size, usually set at 80% |
| Confidence interval | The range of effect sizes consistent with the data |
| Minimum detectable effect | The smallest improvement the test is designed to reliably find |
Sometimes randomization isn't possible: pricing that must be the same for everyone, legal constraints, or too little traffic. The alternatives are rollouts by region with difference-in-differences analysis, before/after analysis with a comparison group, or qualitative research. They're weaker evidence, so state the limitations clearly. Our correlation vs causation guide covers these methods.
A/B testing sits between diagnostic and prescriptive analytics in the four types of data analytics: it tells you what action works. It comes up often in product and marketing analyst interviews, alongside SQL (see our SQL interview questions). A well-documented A/B analysis on a public dataset is also a strong portfolio piece if you're breaking in without experience.
For the statistics behind testing (hypothesis tests, confidence intervals, power), the Statistics with Python Specialization (Michigan) is hands-on and code-based. For the marketing side of experimentation and measurement, see the Marketing Analytics Specialization. Browse all Math & Statistics courses.
How long should an A/B test run? Until it reaches the sample size calculated in advance, and for at least one or two full weekly cycles. Don't stop just because the result looks significant early.
What's a good sample size for an A/B test? It depends on your baseline rate and the smallest effect you care about. Detecting small lifts on low conversion rates needs tens of thousands of users per group, as in our example. Calculate it before starting.
What does p < 0.05 actually mean? If the change had no real effect, you'd see a difference at least this large less than 5% of the time. It isn't the probability that the variant is better, and it says nothing about how big the effect is.
What's the difference between A/B testing and multivariate testing? An A/B test compares whole versions (A vs B, or A vs B vs C). Multivariate testing changes several elements at once to measure each one's effect and how they interact. That needs much more traffic.
A/B testing uses random assignment to measure what a change actually causes. Write the hypothesis and the primary metric first, calculate the sample size before starting, don't peek, then judge both significance and the size of the effect (the confidence interval). Done properly, it replaces opinions about what works with evidence.
A three-course University of Michigan series teaching statistical visualization, inference, and regression modeling in Python, at beginner level.
A six-course specialization teaching marketing analytics skills using Excel, from exploratory data analysis to predictive modeling and forecasting.