Linear and Logistic Regression: Complete Beginner's Guide

Linear regression predicts a number by fitting a straight-line relationship: price = base + rate times area, choosing the line closest to all training points. Logistic regression — despite its name — is a classifier: it squeezes a linear score through the S-shaped sigmoid function into a 0–1 probability.

These two are the workhorses of applied machine learning: simple, fast, interpretable, and strong baselines. Their coefficients say directly how much each feature matters, training takes seconds, and results are easy to explain to non-technical stakeholders.

Key Concepts

• Coefficient: a feature's weight; 15 means each extra unit adds 15 to the prediction.
• Intercept: the prediction when all features are zero.
• Least squares: fitting by minimizing the sum of squared errors.
• Sigmoid: sigma(z) = 1 divided by (1 + e^(−z)); squashes any number into a 0–1 probability.
• Log-odds: logistic regression models log(p divided by (1 − p)) as linear in the features.
• Decision boundary: usually p = 0.5 separates the predicted classes.

Formulas

• Linear regression: prediction = b0 + b1 times x1 + … + bp times xp. The prediction is in the target's units; the bi are learned coefficients. Assumes an approximately linear relationship, independent errors with constant variance.
• Logistic regression: z = b0 + b1 times x1 + …, then p = 1 divided by (1 + e^(−z)). Classify positive when p is at least 0.5. Assumes log-odds change linearly with the features.

Worked Example

Linear case: flats with rooms 1, 2, 3, 4, 5 sold for 20, 35, 50, 65, 80 (lakh rupees). Means: x = 3, y = 50. Slope = (60 + 15 + 0 + 15 + 60) divided by (4 + 1 + 0 + 1 + 4) = 150 divided by 10 = 15. Intercept = 50 − 15 times 3 = 5. Model: price = 5 + 15 times rooms. A 6-room flat: 5 + 90 = 95 lakh rupees. Logistic case: pass model z = −6 + 0.8 times study hours. For 10 hours: z = −6 + 8 = 2. Then p = 1 divided by (1 + e^(−2)); e^(−2) is about 0.1353, so p = 1 divided by 1.1353, about 0.881 — an 88.1 percent pass probability, so predict "pass".

Common Misconceptions

• Logistic regression does regression. A historical name; it outputs class probabilities for classification.
• Linear regression fits only straight lines. With x-squared features it fits curves — "linear" refers to the coefficients.
• A strong fit proves causation. Regression finds correlation, not causation.

Real-World Applications

• Price and demand forecasting: houses, sales, energy use.
• Click-through prediction: probability a user clicks an ad.
• Medical risk scores: disease probability from patient measurements.
• Customer churn: flagging subscribers likely to cancel.

Linear regression predicts numbers by least squares; logistic regression predicts class probabilities through the sigmoid. Both are fast, interpretable, excellent first models — learn to read their coefficients and check their assumptions.