Decision Trees and Random Forests: Complete Beginner's Guide

A decision tree predicts by asking a series of yes/no questions: "Income above 50,000? If yes, is credit history good?" Each internal node tests one feature, branches follow the answers, and leaves give the final prediction. Trees are built by choosing, at each node, the question that best separates examples into pure groups.

A random forest grows hundreds of such trees on random subsets of data and features, then lets them vote (classification) or average (regression). Single trees are intuitive but unstable — tiny data changes reshape them; forests average away that instability. Forests are famously reliable "just works" models for tabular data.

Key Concepts

• Decision node: tests one feature against a threshold.
• Leaf node: gives the final prediction — majority class or average value.
• Split: the chosen feature-plus-threshold dividing examples at a node.
• Impurity: how mixed the classes are at a node; splits aim to reduce it.
• Gini impurity: 1 − sum of class fractions squared; 0 means perfectly pure.
• Entropy: negative sum of (fraction times log base 2 of fraction), in bits.
• Bagging: each tree trains on a bootstrap (random with-replacement) sample.
• Pruning: cutting overgrown branches to fight overfitting.

Formulas

• Gini impurity: Gini = 1 − sum over classes of (pi squared), where pi are unitless class fractions summing to 1. Lower is purer; the tree picks splits with the largest impurity reduction.
• Entropy: H = negative sum of (pi times log2(pi)), measured in bits; 0 for a pure node. Both assume categorical labels and treat all misclassifications as equally bad.

Worked Example

A node holds 10 loan applicants: 6 repaid, 4 defaulted. Gini = 1 − (0.6 squared + 0.4 squared) = 1 − 0.52 = 0.48 — fairly mixed. Split on "income above 50,000": left child gets 5 yes and 1 no, Gini = 1 − (0.694 + 0.028), about 0.278; right child gets 1 yes and 3 no, Gini = 1 − (0.0625 + 0.5625) = 0.375. Both children are purer than 0.48, so the split is useful. Now the forest: 5 trees vote [repay, repay, default, repay, default] — 3 of 5 say repay, so the forest predicts "repay".

Common Misconceptions

• Bigger trees are better. Deep trees memorize training data; pruning or depth limits keep them generalizing.
• A forest is just identical trees. Bootstrap samples plus random feature subsets make trees diverse — that diversity is why voting works.
• Trees handle only categorical data. Numeric features split on thresholds like "age at most 30".

Real-World Applications

• Medical diagnosis: readable diagnostic flowcharts doctors can inspect.
• Customer churn: predicting leavers and revealing the most predictive features.
• Credit approval: interpretable rules with strong accuracy on tabular data.
• Feature importance: ranking which measurements drive predictions.

Decision trees predict by asking impurity-reducing questions measured by Gini or entropy; random forests grow many diverse trees and let them vote. Interpretable, strong on tables, easy to tune — a default choice for structured data.