Type I and Type II Errors in Hypothesis Testing Guide

In hypothesis testing two mistakes are possible. A Type I error means rejecting a null hypothesis that was actually true — a false alarm. A Type II error means failing to reject a null hypothesis that was actually false — a missed detection.

These errors matter because their costs are rarely equal. In a disease screening test, a Type I error (telling a healthy person they are sick) causes worry and extra tests, but a Type II error (telling a sick person they are healthy) can cost a life. Knowing which error you fear more tells you how to design the test.

Key Concepts

• Type I error (false positive): rejecting H₀ when H₀ is true. Its probability is denoted α, the significance level.
• Type II error (false negative): failing to reject H₀ when H₀ is false. Its probability is denoted β.
• Power of a test: 1 − β, the probability of correctly detecting a real effect when it exists. Researchers usually aim for power of at least 0.8.
• The 2×2 truth table: reality (H₀ true / H₀ false) crossed with your decision (reject / fail to reject) gives four cells — two correct decisions and the two errors.
• The α–β trade-off: with a fixed sample size, making α smaller (fewer false alarms) makes β larger (more missed detections), and vice versa. Collecting more data is the only way to shrink both at once.

Formulas

• Power = 1 − β. A test with β = 0.2 has 80% power: it will catch a real effect of the specified size 8 times out of 10.
• Sample size for a mean (approximate): n ≈ ((zα + zβ) × σ / Δ)², where Δ is the smallest difference worth detecting.

Worked Example

A factory tests H₀: "the new machine produces at most 2% defectives" at α = 0.05. Consider what each outcome means:

1. Reality: H₀ true (machine is fine), decision: reject. Type I error — the factory scraps a good machine. Probability: 5%.
2. Reality: H₀ false (machine is faulty), decision: fail to reject. Type II error — faulty products ship to customers.
3. Suppose the test's β = 0.20. Then power = 1 − 0.20 = 0.80: if the machine truly is faulty, the test catches it 80% of the time.
4. Improvement: testing 400 items instead of 100 halves the standard error, raising power without touching α.

Common Misconceptions

• α + β does not equal 1. α assumes H₀ is true, β assumes it is false — they are probabilities under different realities, not complements.
• Power is not about H₀ at all. It answers "if the effect is really there, will we see it?" — a property of the test's sensitivity.
• A non-significant result is not proof of no effect. It may simply be a Type II error from a weak, underpowered test.

Real-World Applications

• Medical screening: tests minimise Type II errors (missed illness), accepting more false alarms, because a missed diagnosis is far costlier.
• Criminal trials: "innocent until proven guilty" minimises Type I errors (convicting the innocent), accepting that some guilty people go free.
• Spam filters: tuned to avoid Type I errors (real mail sent to spam), tolerating a few spam messages slipping through.

Type I error is the false alarm with probability α, Type II error is the missed detection with probability β, and power (1 − β) measures the test's ability to catch real effects. Balance them deliberately — the right balance depends on which mistake hurts more.