Law of Large Numbers: Weak and Strong Laws Explained

The Law of Large Numbers (LLN) says that as you repeat a random experiment more and more times, the average of the results gets closer and closer to the true expected value. Flip a coin 10 times and you might get 7 heads; flip it 10,000 times and the fraction of heads will sit very near 0.5.

It matters because it is the mathematical licence for all of statistics: insurance companies set premiums, casinos set odds, and pollsters trust samples, all because averages stabilise as samples grow. Without the LLN, a sample mean would tell you nothing reliable about the population.

Key Concepts

• Sample mean X̄: the average of n independent observations, X̄ = (X₁ + X₂ + … + Xₙ)/n.
• Weak law: for any fixed tolerance ε > 0, P(|X̄ - μ| > ε) → 0 as n → ∞. The chance of a big miss shrinks to zero.
• Strong law: X̄ → μ almost surely, meaning the averages converge with probability 1, a stronger guarantee than the weak law.
• Chebyshev's inequality: P(|X̄ - μ| ≥ ε) ≤ σ²/(nε²), which proves the weak law and gives a concrete bound for finite n.
• Requirements: the observations must be independent with the same mean μ, and the weak law via Chebyshev needs a finite variance σ².

Formulas

Chebyshev bound for the sample mean: P(|X̄ - μ| ≥ ε) ≤ σ²/(nε²). Expected value of the sample mean: E[X̄] = μ; its variance: Var(X̄) = σ²/n, which is why the bound shrinks as n grows. The law guarantees convergence of the average, not correction of past luck: it does not say tails are "due" after a run of heads.

Worked Example

A fair die has μ = 3.5 and σ² = 35/12 ≈ 2.9167. Roll it n = 1000 times. How likely is the average to miss 3.5 by more than 0.2? Using Chebyshev with ε = 0.2: P(|X̄ - 3.5| ≥ 0.2) ≤ 2.9167/(1000 × 0.04) = 2.9167/40 ≈ 0.0729. So there is at most a 7.3% chance the 1000-roll average falls outside 3.3 to 3.7. With n = 100,000 the bound becomes 0.000729, under 0.1%: the average is pinned near 3.5.

Common Misconceptions

• It does not "balance out" in the short run: after 9 heads in 10 flips, the 11th flip is still 50-50. The LLN works by diluting old results with mountains of new ones, not by forcing compensation.
• Weak vs strong is about the type of convergence: the weak law says large deviations become unlikely; the strong law says the averages themselves settle down with probability 1. Strong implies weak, not the other way round.
• It needs a finite mean: for distributions with no finite mean (like the Cauchy), the LLN fails and averages keep jumping wildly no matter how large n is.

Real-World Applications

• Insurance: with millions of policies, the average claim converges to the expected loss, so premiums can be priced just above it.
• Casinos: each game has a small house edge in expectation; over millions of bets the average profit per bet converges to that edge.
• Polling: larger samples give sample averages that hug the true population value, which is why big polls are trusted.

The Law of Large Numbers promises that averages converge to expectations as n grows, with Chebyshev's bound σ²/(nε²) showing exactly how fast the risk of a big miss shrinks. It is the reason samples work at all: collect enough data, and the average stops being a gamble and starts being a measurement.