Floating Point Arithmetic and Error Analysis: Notes and FAQs

Floating point arithmetic is how computers represent real numbers with finitely many binary digits, so almost every real is rounded to the nearest representable value. IEEE 754 double precision uses 64 bits: 1 sign bit, 11 exponent bits, 52 fraction bits, about 15-16 significant decimal digits. Each operation is correctly rounded: the computed result is the exact result rounded to the nearest floating point number.

Error analysis matters because rounding errors accumulate and can be catastrophically amplified. A numerically unstable algorithm returns garbage for well-posed problems, while a stable one keeps error near the rounding level. Machine epsilon, cancellation, and conditioning let you predict accuracy and choose stable algorithms.

Key Concepts

• Machine epsilon eps = 2^{-52}, about 2.22 x 10^{-16}; the gap between 1 and the next double.
• Rounding model: fl(x op y) = (x op y)(1 + delta) with |delta| <= eps/2 per operation (round to nearest).
• Absolute vs relative error absolute error is |computed - true|; relative error divides by |true|.
• Catastrophic cancellation: subtracting nearly equal numbers leaves mostly rounding error; the relative error of the difference can be enormous.
• Condition number how much a problem amplifies input errors; ill-conditioned problems defeat any algorithm.
• Forward vs backward error: forward error is how wrong the answer is; backward error is how much the input must change to make it exact.

Formulas

• Machine epsilon: eps = 2^{-52} approx 2.22 x 10^{-16}; unit roundoff u = eps/2 approx 1.11 x 10^{-16} bounds each rounding.
• Rounding bound: |fl(x) - x| / |x| <= u for round-to-nearest; every stored number carries at most this relative error.
• Cancellation amplification: for a - b with a approx b, relative error is about u * |a| / |a - b|, blowing up as a - b -> 0.
• Error accumulation: naive summation of n numbers accumulates about n*u relative error; Kahan compensated summation reduces it to about 2u.

Worked Example

Watch double precision compute 0.1 + 0.2. Neither is exactly representable in binary; they store as about 0.1000000000000000055511 and 0.200000000000000011102. Their sum rounds to 0.300000000000000044409, printing as 0.30000000000000004. The relative error is about 1.85 x 10^{-16}, within one unit roundoff: correct rounding, not a bug. The dangerous case is cancellation: with x = 1e-16, compute fl((1 + x) - 1). Doubles near 1 are spaced 2^{-52} (about 2.22e-16) apart, and x is below half that spacing, so 1 + x rounds back to exactly 1, giving 0 instead of 1e-16: 100 percent relative error from one subtraction. The fix is algebraic reformulation, e.g. expm1(x) or log1p(x), computed without the cancellation.

Common Misconceptions

• "0.1 + 0.2 != 0.3 is a bug." It is correct rounding of inexact binary inputs; decimal fractions like 0.1 are infinitely repeating in binary, just as 1/3 repeats in decimal.
• "More precision fixes everything." Higher precision shrinks rounding error, but ill-conditioning or an unstable algorithm still destroys accuracy.
• "Small residual means an accurate answer." A tiny residual only means small backward error; for ill-conditioned problems the forward error can still be large.

Real-World Applications

• Scientific computing stable factorizations (QR, not normal equations) and compensated summation keep accuracy.
• Machine learning: mixed-precision training manages rounding deliberately; softmax is written in cancellation-free form.
• Finance decimal or integer-cents arithmetic avoids binary rounding for money.
• Graphics and geometry: robust predicates use exact or adaptive arithmetic so rounding cannot flip an inside/outside test.

Floating point rounds every operation to 16 decimal digits. The rounding model bounds each step, cancellation is the great amplifier, and the condition number separates problem difficulty from algorithm quality.