Statistics with Python Specialization (Michigan)
A three-course University of Michigan series teaching statistical visualization, inference, and regression modeling in Python, at beginner level.
A p-value is the probability of seeing a result at least as extreme as yours if there were actually no real effect (the "null hypothesis"). A small p-value means your data would be surprising in a world with no effect, which is evidence against that world. It is not the probability that your hypothesis is true, and it says nothing about how big or important the effect is.
P-values appear in every A/B test report, research paper and statistics course, and they're misread constantly, even by experienced analysts. The idea itself is simple. It's usually explained badly.
A friend hands you a coin and claims it's fair. You're suspicious, so you flip it 100 times and get 60 heads.
Is the coin biased? Maybe. But a fair coin doesn't give exactly 50 heads every time either. Randomness alone produces some variation. The real question is:
If the coin were fair, how often would I see a result at least this far from 50?
That "how often" is the p-value. For 60 or more heads, or 40 or fewer (equally extreme in the other direction), an exact calculation gives p ≈ 0.057. A fair coin would produce a result this lopsided about 5.7% of the time.
That's somewhat unusual, but not rare. Using the common 5% threshold, you would not call this significant. You don't have strong enough evidence that the coin is biased.
Now flip it 1,000 times and get 600 heads, the same 60% rate. A fair coin would almost never do that: the p-value is far below 0.000001. Same proportion, much more data, so much stronger evidence.
Lesson one: p-values depend heavily on sample size. More data can reveal smaller effects.
It's a bit like a courtroom: assume innocence, then ask whether the evidence would be very unlikely if the defendant were innocent.
These are the misreadings that cause real mistakes.
| Common claim | Why it's wrong |
|---|---|
| "p = 0.03 means there's a 3% chance the result is due to chance." | The p-value assumes chance is the only explanation and asks how surprising the data is. It isn't the probability that chance caused it. |
| "p = 0.03 means a 97% chance our hypothesis is true." | P-values don't give the probability of any hypothesis being true. |
| "p < 0.05 means the effect is big / important." | With enough data, tiny, worthless effects become "significant". |
| "p = 0.20 proves there's no effect." | It only means you didn't find strong evidence. The test may have been too small to detect a real effect. |
| "p = 0.049 and p = 0.051 are fundamentally different." | They're almost identical evidence. The 0.05 line is a convention, not a law of nature. |
The American Statistical Association took the unusual step of publishing a formal statement on p-values in 2016, largely because these misinterpretations were so widespread.
An A/B test on a checkout page gives:
What you can honestly say:
What you shouldn't say: "There's a 99.8% chance the new checkout is better."
A p-value answers "is there something here?" A business decision needs "how much, and is it worth it?" So always report:
A result can be highly significant and commercially trivial, like a 0.01-point lift measured on 10 million users. It can also be non-significant but promising, like a large lift measured on a sample too small to be sure.
If you test enough things, something will come out "significant" by chance. At the 5% level, about 1 in 20 tests of pure noise will cross the line. Common ways this happens:
The defenses: decide the hypothesis, the primary metric and the sample size before looking at the data, report everything you tested, and treat surprising subgroup findings as ideas to test again, not as conclusions.
P-values sit inside a bigger toolkit: sampling, confidence intervals and hypothesis testing. Statistics with Python Specialization (Michigan) teaches inference with hands-on code, and Introduction to Statistics (Stanford) builds the intuition conceptually. If you're heading toward ML, see how much math you really need. Browse all Math & Statistics courses.
What is a good p-value? There isn't a "good" p-value. There's a threshold chosen in advance, usually 0.05 and sometimes 0.01 when false positives are costly. Report the actual value and the effect size, not just "significant or not".
Why 0.05? Mostly convention, going back to early 20th-century statistics. It's a reasonable default for many business decisions, but not a law. The right threshold depends on the cost of a false positive.
What's the difference between a p-value and a confidence interval? The p-value tells you how surprising the data is under "no effect". The confidence interval gives a range of plausible effect sizes. The interval is usually more useful for decisions because it shows magnitude.
Can a p-value be exactly zero? No. Very small p-values are often displayed as 0.000 or "< 0.001", but they're never truly zero.
A p-value measures how surprising your data would be if nothing were going on. Small values are evidence against "no effect". It isn't the probability your hypothesis is true, and it doesn't measure importance. Decide your test in advance, report effect sizes and confidence intervals alongside p-values, and be most skeptical of "significant" results you found by searching.
A three-course University of Michigan series teaching statistical visualization, inference, and regression modeling in Python, at beginner level.
A beginner-level Stanford course covering descriptive stats, probability, regression, and hypothesis tests, taught across 12 self-paced modules.