Model validation

What it is

Model validation is the independent assessment of a model by people who did not build it. It asks whether the model is fit for purpose and whether its limits are understood. Supervisory guidance in Europe, the United Kingdom and the United States describes similar elements:

• conceptual soundness: the theory, assumptions and methods suit the use;
• data: inputs are complete, accurate and representative;
• implementation: the code and systems compute what the documentation says;
• performance: outcomes are tested through backtesting, benchmarking and sensitivity analysis;
• use: the model is used within its scope, and limitations are known.

Validation happens before a model goes live and then periodically, with frequency set by the model’s tier. Each finding gets a severity and a remediation date.

Why it matters

Firms use models to set capital, provisions, prices, limits and hedges. A flawed model can misstate all of them at once. Validation is the main control over model risk, and supervisors expect it to be independent and able to stop or restrict a model. For capital models, supervisors also review validation work during model approvals and inspections.

Worked example

Illustrative numbers. A validation team reviews a retail PD model three years after development.

• Ranking: the area under the ROC curve (AUC) was 0.78 at development and is now 0.71. The fall is moderate, and the model still separates good and bad borrowers.
• Calibration: grade 4 predicted a 2.0 percent default rate; 3,000 borrowers produced 93 defaults, an observed rate of 3.1 percent. A binomial test rejects the predicted rate at the 99 percent level.
• Stability: the population stability index for the score is 0.28, above the 0.25 trigger, showing that the portfolio has shifted since development.

The team raises a high-severity finding on calibration, requires recalibration within six months and asks for a temporary conservative adjustment in the meantime.

Who uses it

• Model Validation Analysts perform the reviews and write validation reports.
• Credit Risk Model Developers respond to findings and remediate models.
• Market Risk Analysts and ALM Analysts work with validators on risk measurement and behavioural models.
• Data Scientists (Banking) see their machine learning models validated under the same framework.
• Risk Advisory Consultants support validation work and framework design.

Common confusions

• Validation is wider than backtesting. A model can pass a backtest and still rest on weak assumptions.
• Independence is about incentives and reporting lines. Validators can sit in the same risk function as long as they do not build or own the model.
• Monitoring by the model owner complements validation and does not replace it.
• A validated model is not error-free. Validation documents known limits and how they are managed.