Entropy and Information: Complete Beginner's Guide

In information theory, entropy measures the uncertainty — equivalently, the information content — of a random source. A fair coin toss has high entropy because the outcome is unpredictable; a coin landing heads 99 percent of the time has low entropy because you can guess the result. Claude Shannon defined it in 1948, founding the mathematical theory of communication.

It matters because entropy sets the fundamental limits of data handling: the smallest possible average size of a compressed file, the redundancy an error-correcting code needs, and the information a machine-learning split gains. Every ZIP file rests on it.

Key Concepts

• Bit: the unit of information — what you learn from one yes/no question with equally likely answers.
• Probability distribution: the probabilities p(x) of each possible outcome. Entropy is computed from these, not from any single outcome.
• Surprise: an outcome with probability p carries −log₂(p) bits of surprise — rare events surprise more.
• Entropy H(X): the average surprise over all outcomes — the expected information per symbol.
• Mutual information: how much knowing one variable reduces uncertainty about another; the basis of information gain in decision trees.
• Channel capacity: the maximum rate of reliable communication over a noisy channel.

Formulas

• Shannon entropy: H(X) = −Σ p(x)·log₂ p(x), summed over all outcomes. Unit: bits per symbol (base 2 gives bits; natural log would give nats).
• Binary entropy (probabilities p and 1−p): H = −p·log₂ p − (1−p)·log₂(1−p). Peaks at 1 bit when p = 0.5, falls to 0 at p = 0 or 1.
• Channel capacity (Shannon–Hartley): C = B·log₂(1 + S/N), with bandwidth B in hertz and signal-to-noise ratio S/N dimensionless. C is in bits per second.
• Assumes independent symbols; correlated sources need conditional entropy.

Worked Example

Compute the entropy of a fair coin and of a coin landing heads with probability 0.9.

1. Fair coin: H = −(0.5·log₂ 0.5 + 0.5·log₂ 0.5) = −(−0.5 − 0.5) = 1 bit. Maximum uncertainty.
2. Biased coin: log₂ 0.9 ≈ −0.152, log₂ 0.1 ≈ −3.322. H = −(0.9 × −0.152 + 0.1 × −3.322) = −(−0.1368 − 0.3322) = 0.469 bits.

The biased coin carries under half a bit per toss — you learn little, since heads was the safe guess anyway. Lower entropy means better compressibility: 0.469 bits per symbol is the theoretical floor for such a file.

Common Misconceptions

• "Entropy is the same as file size." File size is what you have; entropy is the minimum average bits per symbol you could achieve — the target compression aims at but never beats.
• "More data always means more information." Redundant data adds bytes without entropy; ten copies of one email hold roughly one email's information.
• "Encrypted data has more information." Good encryption looks random, raising entropy per symbol — unpredictability to an eavesdropper, not more meaningful content.

Real-World Applications

• Compression: ZIP, PNG and MP3 shrink files toward the Shannon limit by exploiting low entropy.
• Error correction: capacity tells engineers how much redundancy guarantees reliability — used in 5G, Wi-Fi and deep-space links.
• Machine learning: decision trees split on the feature with the highest information gain.

Entropy quantifies uncertainty as H(X) = −Σ p(x)·log₂ p(x): maximum when outcomes are equally likely, zero when the result is certain. It draws the hard lines — the smallest file, the fastest reliable channel — that all of computing respects.