Hypothesis Testing Framework: Steps, p-Value and Errors

A hypothesis test is a rule-based way to decide whether sample data supports a claim about a whole population. You follow a fixed procedure ending in a clear verdict: reject the claim, or keep it for now because the evidence is too weak.

This matters because data-driven decisions carry real consequences — drug trials, factory quality checks, policy evaluations. Hypothesis testing puts a number on how surprising the data would be if the claim were false, so decisions rest on evidence rather than opinion.

Key Concepts

• Population and sample: the population is everyone you care about; the sample is the smaller group you actually measure.
• Parameter vs statistic: a parameter (μ, p, σ) describes the population and is usually unknown; a statistic (x̄, p̂, s) is computed from the sample.
• Null hypothesis (H₀): the default claim, always written with an equality sign, for example μ = 100.
• Alternative hypothesis (H₁): what you conclude if the data contradicts H₀, for example μ ≠ 100.
• Significance level (α): your pre-set tolerance for a false alarm, most commonly 0.05.
• Test statistic: a number computed from the sample measuring how far the data sits from what H₀ predicts.
• p-value: the probability of data this extreme, or more extreme, if H₀ were true.

The standard steps are: (1) state H₀ and H₁; (2) choose α first; (3) compute the test statistic; (4) find the p-value; (5) reject H₀ if p < α; (6) restate the conclusion plainly.

Formulas

• z-test for a mean (σ known): z = (x̄ − μ₀) / (σ / √n). Compares the sample mean with the claimed value μ₀.
• t-test for a mean (σ unknown): t = (x̄ − μ₀) / (s / √n), with degrees of freedom df = n − 1. Uses the sample standard deviation s instead of σ.
• Decision rule: reject H₀ when p-value < α. Smaller p means stronger evidence against H₀.

Assumptions: the sample should be random; for small samples the population should be approximately normal.

Worked Example

Question: is a coin fair? Toss it 100 times and get 60 heads.

1. Hypotheses: H₀: p = 0.5 (fair coin); H₁: p ≠ 0.5; α = 0.05.
2. Sample proportion: p̂ = 60 / 100 = 0.60.
3. Standard error under H₀: √(0.5 × 0.5 / 100) = √0.0025 = 0.05.
4. Test statistic: z = (0.60 − 0.50) / 0.05 = 2.00.
5. p-value (two-sided): 2 × P(Z > 2.00) = 2 × 0.0228 = 0.0456.
6. Decision: 0.0456 < 0.05, so reject H₀.
7. Conclusion: the data suggests the coin is biased — though a fair coin could still produce this result about 4.6% of the time.

Common Misconceptions

• The p-value is not the probability that H₀ is true. It is the probability of the data given H₀. A p-value of 0.04 does not mean there is a 96% chance the alternative is true.
• "Fail to reject H₀" does not prove H₀. It only means the evidence was not strong enough — perhaps the sample was simply too small.
• Choosing α after seeing the data invalidates the test. Setting α just below the observed p-value guarantees a "significant" result, which is cheating, not statistics.

Real-World Applications

• Clinical trials: testing whether a new drug beats a placebo before regulators approve it.
• Quality control: checking each batch of parts against the specification.
• A/B testing: comparing two page designs to see whether the new one genuinely raises the click rate.

Hypothesis testing turns "does this claim hold?" into a disciplined procedure with a quantified answer. Master this framework once, and every specific test (z, t, chi-square, F) becomes the same logic in different formulas.