Least Squares Curve Fitting: Line of Best Fit Explained

Least squares curve fitting is a method for drawing the best possible line or curve through a set of scattered data points. When measurements are noisy, the points never sit exactly on a line, so we need a rule for choosing the curve that fits them best.

This matters everywhere data is collected: predicting exam scores from study hours, fitting a trend to stock prices, or calibrating a sensor. Least squares turns the messy job of "drawing a good line" into a precise mathematical calculation with a unique answer.

Key Concepts

• Residual: the vertical gap between an actual data point and the fitted curve: rᵢ = yᵢ − f(xᵢ).
• Sum of squared residuals (SSE): r₁² + r₂² + … + rₙ². Least squares chooses the curve that makes this sum smallest.
• Line of best fit: the straight line y = a + bx minimising SSE.
• Normal equations: the equations solved to find the best a and b.
• Regression: another name for fitting, especially lines, to data.

Formulas

For the line y = a + bx fitted to n points (xᵢ, yᵢ):

• Slope: b = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²
• Intercept: a = ȳ − bx̄, where x̄ and ȳ are the means.

Assumptions: errors are random, roughly equal-sized, and the straight-line model is appropriate. Outliers can pull the fitted line far off because squaring makes big residuals matter a lot.

Worked Example

Fit a line to the points (1, 2), (2, 4), (3, 5):

1. Means: x̄ = (1+2+3)/3 = 2, ȳ = (2+4+5)/3 = 11/3 ≈ 3.6667
2. Numerator: (1−2)(2−3.6667) + (2−2)(4−3.6667) + (3−2)(5−3.6667) = (−1)(−1.6667) + 0 + (1)(1.3333) = 1.6667 + 1.3333 = 3.0
3. Denominator: (1−2)² + (2−2)² + (3−2)² = 1 + 0 + 1 = 2
4. Slope b = 3.0/2 = 1.5
5. Intercept a = 3.6667 − 1.5·2 = 3.6667 − 3 = 0.6667

The line of best fit is y = 0.6667 + 1.5x. Check: at x = 2 it predicts 3.6667, exactly the mean ȳ — the fitted line always passes through (x̄, ȳ).

Common Misconceptions

• "The best-fit line passes through most points." It need not pass through any point at all; it minimises total squared error instead.
• "It predicts well far outside the data." Extrapolation beyond the data range is risky and often wrong.
• "A tight fit proves causation." Least squares only measures association, never cause and effect.

Real-World Applications

• Economists fit demand curves to price and sales data for pricing decisions.
• Doctors estimate drug dose-response relationships from trial measurements.
• Engineers calibrate instruments by fitting lines to known reference readings.

Least squares fitting chooses the line that makes the total squared error as small as possible, with a simple closed-form formula for slope and intercept. It is the foundation of regression analysis, used in every field that collects and models data.