Jacobi and Gauss-Seidel Iterative Methods Explained

Jacobi and Gauss-Seidel solve large linear systems iteratively: instead of transforming the matrix in one shot like Gaussian elimination, they start from a guess and improve it sweep by sweep. Each equation is solved for its own diagonal unknown using current values of the others. Jacobi uses only old values from the previous sweep; Gauss-Seidel immediately reuses each freshly computed value within the same sweep.

Direct elimination is too slow and memory-hungry for the giant systems from physics simulations — millions of equations from discretized heat or fluid flow. Iterative methods need only the matrix and a few vectors, and can stop early once accurate enough, often long before a direct method would finish.

Key Concepts

• Diagonal dominance: each diagonal entry exceeds the sum of the other magnitudes in its row — a simple sufficient condition for both methods to converge.
• Jacobi update: all new values are computed from the old vector only, so the updates can be done in parallel.
• Gauss-Seidel update: each new value is used immediately for the remaining unknowns in the same sweep, which usually converges about twice as fast as Jacobi.
• Stopping rule: stop when the change between successive sweeps, or the residual ||b − Ax||, drops below the tolerance.

Formulas

• Jacobi: x_i^(k+1) = (b_i − Σ_j≠i a_ij x_j^(k)) / a_ii, for each i.
• Gauss-Seidel: x_i^(k+1) = (b_i − Σ_j<i a_ij x_j^(k+1) − Σ_j>i a_ij x_j^(k)) / a_ii. Note the already-updated values x_j^(k+1) for j < i.
• Convergence guarantee: both converge for strictly diagonally dominant matrices (and for symmetric positive definite matrices in the Gauss-Seidel case). All formulas require nonzero diagonal entries.

Worked Example

Solve 4x + y = 9 and x + 3y = 8 by Gauss-Seidel from (0, 0). The matrix is strictly diagonally dominant (|4| > |1|, |3| > |1|), so convergence is guaranteed. Sweep 1: x₁ = 9/4 = 2.25; y₁ = (8 − 2.25)/3 = 1.916667 (fresh x₁ used immediately). Sweep 2: x₂ = (9 − 1.916667)/4 = 1.770833; y₂ = (8 − 1.770833)/3 = 2.076389. Sweep 3: x₃ = (9 − 2.076389)/4 = 1.730903; y₃ = (8 − 1.730903)/3 = 2.089699. The exact solution x = 1.727273, y = 2.090909 (check: 4(1.727273) + 2.090909 ≈ 9) is visibly being homed in on.

Common Misconceptions

• "Gauss-Seidel is always faster than Jacobi." Usually, but not guaranteed — for some matrices Jacobi converges while Gauss-Seidel diverges, and vice versa. Diagonal dominance is the safe zone for both.
• "Iterative methods work on any system." They need the convergence conditions; without them the iterates can oscillate or blow up. Direct methods have no such restriction (apart from singularity).
• "The starting guess does not matter." When the method converges, any start reaches the same answer, but a good start (like the previous time step in a simulation) can cut the iteration count dramatically.

Real-World Applications

• Heat transfer: steady-state temperatures on a fine grid satisfy huge systems solved iteratively.
• Power systems: load-flow analysis repeatedly solves large nonlinear-ish systems with Gauss-Seidel-type sweeps.
• Image processing: deblurring and denoising solve large systems where early stopping also filters noise.
• Weather and ocean models: implicit time steps need fast approximate solves of enormous sparse systems.

Jacobi updates every unknown from the old values while Gauss-Seidel reuses fresh values immediately, making it typically twice as fast. Both are guaranteed to converge for diagonally dominant systems and are the methods of choice when linear systems grow too large for direct elimination.