P-Values Explained Without the Jargon

P-Values Explained Without the Jargon

A p-value is the probability of seeing a result at least as extreme as yours if there were actually no real effect (the "null hypothesis"). A small p-value means your data would be surprising in a world with no effect, which is evidence against that world. It is not the probability that your hypothesis is true, and it says nothing about how big or important the effect is.

P-values appear in every A/B test report, research paper and statistics course, and they're misread constantly, even by experienced analysts. The idea itself is simple. It's usually explained badly.

Start With a Coin

A friend hands you a coin and claims it's fair. You're suspicious, so you flip it 100 times and get 60 heads.

Is the coin biased? Maybe. But a fair coin doesn't give exactly 50 heads every time either. Randomness alone produces some variation. The real question is:

If the coin were fair, how often would I see a result at least this far from 50?

That "how often" is the p-value. For 60 or more heads, or 40 or fewer (equally extreme in the other direction), an exact calculation gives p ≈ 0.057. A fair coin would produce a result this lopsided about 5.7% of the time.

That's somewhat unusual, but not rare. Using the common 5% threshold, you would not call this significant. You don't have strong enough evidence that the coin is biased.

Now flip it 1,000 times and get 600 heads, the same 60% rate. A fair coin would almost never do that: the p-value is far below 0.000001. Same proportion, much more data, so much stronger evidence.

Lesson one: p-values depend heavily on sample size. More data can reveal smaller effects.

The Logic in Three Steps

  1. Assume nothing is going on (the null hypothesis: the coin is fair, the new checkout makes no difference).
  2. Calculate how surprising your data would be under that assumption. That's the p-value.
  3. If it's very surprising (below a threshold chosen in advance, usually 0.05), reject the "nothing is going on" assumption. The result is called statistically significant.

It's a bit like a courtroom: assume innocence, then ask whether the evidence would be very unlikely if the defendant were innocent.

What a P-Value Is NOT

These are the misreadings that cause real mistakes.

Common claim Why it's wrong
"p = 0.03 means there's a 3% chance the result is due to chance." The p-value assumes chance is the only explanation and asks how surprising the data is. It isn't the probability that chance caused it.
"p = 0.03 means a 97% chance our hypothesis is true." P-values don't give the probability of any hypothesis being true.
"p < 0.05 means the effect is big / important." With enough data, tiny, worthless effects become "significant".
"p = 0.20 proves there's no effect." It only means you didn't find strong evidence. The test may have been too small to detect a real effect.
"p = 0.049 and p = 0.051 are fundamentally different." They're almost identical evidence. The 0.05 line is a convention, not a law of nature.

The American Statistical Association took the unusual step of publishing a formal statement on p-values in 2016, largely because these misinterpretations were so widespread.

A Business Example

An A/B test on a checkout page gives:

  • Conversion: 4.00% (old) vs 4.55% (new)
  • p ≈ 0.002
  • 95% confidence interval for the lift: +0.20 to +0.90 percentage points

What you can honestly say:

  • "If the new checkout had no effect, a difference this large would be very unlikely (p ≈ 0.002). We have strong evidence it helps."
  • "The lift is plausibly between +0.2 and +0.9 points." This is the part the business actually needs.

What you shouldn't say: "There's a 99.8% chance the new checkout is better."

Why You Should Always Report the Effect Size Too

A p-value answers "is there something here?" A business decision needs "how much, and is it worth it?" So always report:

  • The effect size: +0.55 points, +14% relative, $X per month
  • A confidence interval: the range of plausible effect sizes
  • The p-value, as supporting evidence rather than the headline

A result can be highly significant and commercially trivial, like a 0.01-point lift measured on 10 million users. It can also be non-significant but promising, like a large lift measured on a sample too small to be sure.

P-Hacking: How Good Analysts Fool Themselves

If you test enough things, something will come out "significant" by chance. At the 5% level, about 1 in 20 tests of pure noise will cross the line. Common ways this happens:

  • Checking many metrics and reporting the one that "worked"
  • Slicing data by segment until a significant subgroup appears
  • Stopping an experiment the moment p drops below 0.05
  • Trying several outlier rules or model variants and keeping the best-looking one

The defenses: decide the hypothesis, the primary metric and the sample size before looking at the data, report everything you tested, and treat surprising subgroup findings as ideas to test again, not as conclusions.

Where P-Values Show Up in Data Work

  • A/B tests and experiments: is the difference real?
  • Regression output: is a coefficient distinguishable from zero? Remember that significance still doesn't mean causation.
  • Comparing groups: t-tests, chi-square tests and so on
  • Research papers and vendor claims, where reading them critically is a useful skill in itself

What to Learn Next

P-values sit inside a bigger toolkit: sampling, confidence intervals and hypothesis testing. Statistics with Python Specialization (Michigan) teaches inference with hands-on code, and Introduction to Statistics (Stanford) builds the intuition conceptually. If you're heading toward ML, see how much math you really need. Browse all Math & Statistics courses.

Frequently Asked Questions

What is a good p-value? There isn't a "good" p-value. There's a threshold chosen in advance, usually 0.05 and sometimes 0.01 when false positives are costly. Report the actual value and the effect size, not just "significant or not".

Why 0.05? Mostly convention, going back to early 20th-century statistics. It's a reasonable default for many business decisions, but not a law. The right threshold depends on the cost of a false positive.

What's the difference between a p-value and a confidence interval? The p-value tells you how surprising the data is under "no effect". The confidence interval gives a range of plausible effect sizes. The interval is usually more useful for decisions because it shows magnitude.

Can a p-value be exactly zero? No. Very small p-values are often displayed as 0.000 or "< 0.001", but they're never truly zero.

Bottom Line

A p-value measures how surprising your data would be if nothing were going on. Small values are evidence against "no effect". It isn't the probability your hypothesis is true, and it doesn't measure importance. Decide your test in advance, report effect sizes and confidence intervals alongside p-values, and be most skeptical of "significant" results you found by searching.

Enjoyed this article?

Share it with your network

Listings related to P-Values Explained Without the Jargon