P-Values and Significance Levels: Statistics Guide

The p-value is the probability of getting results at least as extreme as yours, assuming the null hypothesis is true. The significance level α (usually 0.05) is the threshold you fix in advance: if p < α, you reject H₀; otherwise you do not.

These two numbers matter because they are the most quoted — and most misunderstood — quantities in science. Headlines like "study finds significant link" rest entirely on a p-value crossing α. Understanding what that crossing does and does not mean protects you from overclaiming.

Key Concepts

• p-value: P(data this extreme or more | H₀ true). Small p means the data is surprising under H₀.
• Significance level α: your pre-chosen tolerance for Type I error — commonly 0.05, sometimes 0.01 in high-stakes fields.
• "Statistically significant": simply means p < α. It says nothing about whether the effect is large or important.
• Exact p-values beat stars: reporting p = 0.043 is more informative than just saying "significant at 5%".
• p is continuous evidence: p = 0.049 and p = 0.051 are nearly identical evidence, though one crosses the line and the other does not.

How It Works

1. Compute the test statistic from your sample (z, t, χ², or F as appropriate).
2. Find the tail area: the p-value is the area under H₀'s distribution beyond your statistic — both tails for a two-sided test, one tail for a one-sided test.
3. Compare with α: p < α → reject H₀; p ≥ α → fail to reject H₀.
4. Interpret honestly: a tiny p with a huge sample can flag a trivially small effect as "significant" — always look at the effect size too.
5. Never move the goalpost: α must be chosen before seeing the data, or the error guarantee is meaningless.

Worked Example

A two-sided z-test gives z = 1.88. What do we conclude?

1. Tail probability: P(Z > 1.88) ≈ 0.0301 from the normal table.
2. Two-sided p-value: 2 × 0.0301 = 0.0602 ≈ 0.060.
3. At α = 0.05: 0.060 > 0.05, so fail to reject H₀ — not significant.
4. At α = 0.10: 0.060 < 0.10, so reject H₀ — significant.
5. Lesson: the same data, the same p-value, but a different verdict depending on the pre-chosen α. This is exactly why α must be fixed before the analysis — otherwise you could always pick the threshold that gives the answer you want.

Common Misconceptions

• p is not P(H₀ is true). p = 0.03 does not mean there is a 3% chance the null is true or a 97% chance the alternative is true. It is a statement about the data, assuming H₀.
• "Almost significant" is not a finding. p = 0.06 is not "a trend toward significance" — it is a non-rejection. Treat the line as a line.
• Significant ≠ important. With 100,000 observations, a meaningless 0.1% difference can be "highly significant". Always pair p-values with effect sizes and confidence intervals.

Real-World Applications

• Clinical trials: regulators typically demand p < 0.05 on primary outcomes before approving a drug.
• A/B testing: product teams ship the new design only when the conversion lift clears the significance bar.
• Polling: a candidate "leading" by 2 points with p = 0.20 is statistically tied — responsible analysts say so.
• Sports analytics: testing whether a player's improved average is real or a hot streak within normal variation.

The p-value measures surprise under H₀; α is the surprise threshold you commit to in advance. Respect both for what they are — evidence summaries, not truth certificates — and they will keep your conclusions honest.