Model Validation and Sensitivity Analysis Explained

Model validation checks whether a model matches reality, using data the model has never seen. Sensitivity analysis asks how much the predictions change when inputs or assumptions change.

An unvalidated model is just an opinion with equations, and one whose answers swing wildly with tiny input changes cannot guide decisions. Validation and sensitivity analysis are what make modelling honest.

Key Concepts

• Validation — testing the model against independent data; distinct from calibration (fitting the model to data).
• Verification — checking the model is solved correctly (no coding or algebra errors). Verification asks "did we solve the equations right?"; validation asks "are these the right equations?"
• Sensitivity — how much an output changes per unit change in an input parameter.
• Identifiability — whether the data can pin down each parameter, or whether several parameter sets fit equally well.
• Uncertainty quantification — propagating input uncertainty through the model for honest error bars on predictions.

How It Works

A solid validation and sensitivity workflow runs like this:

1. Split your data. Fit (calibrate) parameters on one dataset; test predictions on a separate one. Testing on the training data proves nothing.
2. Choose error metrics. Compare predictions with observations using honest measures: mean absolute error, root mean square error, or relative error — never just "looks close".
3. Check residuals. Plot errors against inputs: random scatter is good; patterns mean the model structure is wrong.
4. Run one-at-a-time sensitivity. Vary each parameter ±10% while holding others fixed; rank parameters by their effect on the key output.
5. Run global sensitivity. Vary all parameters together (e.g. random sampling) to catch interactions that one-at-a-time tests miss.
6. Stress-test assumptions. Deliberately break each one — what if growth is not logistic? what if mixing is not uniform? — and see whether conclusions survive.
7. Report limits. Document where the model was tested, where it failed, and the range of inputs over which you trust it.

Worked Example

A logistic population model N(t) = K/(1 + Ae^(−rt)) was fitted to fishery data giving r = 0.4/yr, K = 100,000. Validate and test sensitivity of the 5-year forecast.

1. Step 1: Validation — hold out the last 2 years of catch data before fitting. The fitted model predicts those years within 8% error: acceptable.
2. Step 2: Residuals show no trend with population size, so the logistic structure looks reasonable.
3. Step 3: Sensitivity — raise r by 10% (0.44): the 5-year forecast rises by about 12%. Raise K by 10% (110,000): the forecast rises by about 9%.
4. Step 4: The forecast is moderately sensitive to r. Honest report: "5-year forecast ≈ 78,000 fish, ±15% given parameter uncertainty" — management should keep a safety margin.

Common Misconceptions

• "A good fit means a valid model." Fitting well to training data is easy — an overcomplicated model can fit anything. Validity is proven only on unseen data.
• "Sensitivity analysis is optional polish." It is core. If your conclusion flips when a parameter moves within its plausible range, you do not have a conclusion.
• "Validated once means valid forever." Conditions change — new variants, new behaviour, new environments. Validation has an expiry date.

Real-World Applications

• Climate models — validated by "hindcasting": predicting past climate from earlier data before forecasting the future.
• Drug trials — pharmacokinetic models are validated on separate patient groups before dosing guidelines are set.
• Financial regulation — banks must validate risk models on historical crises and stress-test them against hypothetical ones.

Validation proves the model matches reality; sensitivity analysis proves the conclusions are robust. Together they turn curve-fitting into a trustworthy basis for decisions: never trust a model you have not tested, nor a conclusion you have not wobbled.