How Brier-Score Algorithms Audit Probability-of-Default Forecasts: Proper Scoring, Calibration, Resolution, Base Rates, Drift and Recalibration Tests

Reader question: A credit model says one borrower has a 2% one-year probability of default and another has 12%. After the year ends, how do we test whether the probability numbers themselves were good rather than merely whether the ranking looked sensible?

The Brier score is one answer. For binary outcomes it measures the mean squared distance between forecast probabilities and what actually happened. Because it is a proper scoring rule, it rewards probability forecasts that are both honest and numerically close to realised frequencies over repeated cases.

This article owns the probability-forecast scoring problem for credit default: predicted PDs + realised default indicators + a defined observation horizon → Brier score, benchmark skill, calibration/resolution diagnostics, confidence checks and recalibration triggers.

It does not own loan approval policy, pricing, capital requirements or personalised lending decisions. A probability forecast can be statistically well calibrated and still be unsuitable for a particular decision threshold, regulatory use or individual case.

This is public mathematical and computational education. It is not credit advice and should not be used to infer any person’s creditworthiness.

1. Classification accuracy is the wrong first question for probabilities

Suppose a model predicts a 4% PD for 10,000 comparable loans. If 400 eventually default, the model may be well calibrated even though a threshold classifier that labels every loan “non-default” would be 96% accurate.

Accuracy has discarded the probability.

The probability question is different:

When the model says 4%, do outcomes occur about 4% of the time in comparable cases?

2. The binary Brier score

For case i, let:

• p_i = predicted probability of default, between 0 and 1;
• y_i = realised outcome, 1 for default and 0 for non-default.

The binary Brier score is:

BS = (1/N) Σ_i=1^N (p_i − y_i)².

Under this binary convention, 0 is perfect and larger values are worse.

3. A single observation teaches very little

If a model assigns 10% PD and the borrower defaults, the squared error is:

(0.10 − 1)² = 0.81.

If the borrower survives, the squared error is:

(0.10 − 0)² = 0.01.

That does not prove the 10% forecast was wrong after one case. A genuine 10% event is supposed to occur occasionally.

Probability verification requires repeated comparable forecasts.

4. Why the Brier score is a proper scoring rule

A proper scoring rule is designed so that, in expectation, a forecaster minimises the score by reporting the probability it actually believes under the model.

If the true event probability is q, the expected binary Brier loss from reporting p is:

q(1−p)² + (1−q)p².

Expanding gives:

q − 2qp + p².

Differentiating with respect to p:

d/dp = 2p − 2q.

The minimum occurs at:

p = q.

This mathematical property is why the Brier score is useful for auditing stated probabilities rather than only hard classifications.

5. A simple portfolio example

Consider four forecasts:

• Loan A: 2%, outcome 0;
• Loan B: 5%, outcome 0;
• Loan C: 20%, outcome 1;
• Loan D: 40%, outcome 0.

The squared errors are:

• A: 0.0004;
• B: 0.0025;
• C: 0.6400;
• D: 0.1600.

The mean is:

BS = (0.0004 + 0.0025 + 0.6400 + 0.1600)/4 ≈ 0.2007.

The number has no useful interpretation by itself until it is compared with a benchmark and decomposed.

6. The base-rate benchmark

A simple reference forecast assigns every case the portfolio default rate:

p_ref = ȳ.

If 3% of comparable observations default, a constant 3% forecast can achieve a surprisingly small Brier score simply because defaults are rare.

This is why rare-event portfolios require a benchmark.

7. Brier skill score

A common comparison is:

BSS = 1 − BS_model / BS_reference.

Then:

• BSS > 0 means the model beats the chosen reference;
• BSS = 0 means no improvement;
• BSS < 0 means the model is worse.

The benchmark must be stated. A skill score against the in-sample base rate is not the same as one against an external or time-matched reference forecast.

8. Murphy-style decomposition

For grouped probability forecasts, the Brier score can be decomposed conceptually into:

Brier score = reliability − resolution + uncertainty.

The three terms answer different questions.

• Reliability: do predicted probabilities match observed frequencies within forecast groups?
• Resolution: does the model separate cases into groups whose observed default rates differ meaningfully from the overall base rate?
• Uncertainty: how inherently variable is the binary outcome given the overall base rate?

9. Reliability is calibration

Suppose a group of 1,000 exposures all receive PDs near 8%, but 140 default.

The observed rate is 14%, so the model is underpredicting risk in that group.

A reliability term penalises the gap between forecast probability and empirical outcome frequency.

Diagnostic: plot forecast bins against observed default rates with uncertainty intervals.

10. Resolution is different from calibration

A model that assigns every borrower the same 5% PD can be perfectly calibrated if the total default rate is 5%.

But it has no resolution: it has not separated lower-risk from higher-risk cases.

Conversely, a model can rank borrowers very well while systematically assigning probabilities that are too high or too low.

This is why calibration and discrimination must be reported separately.

11. AUC does not replace the Brier score

The area under the ROC curve measures ranking discrimination. It asks how often a randomly selected defaulter receives a higher risk score than a randomly selected non-defaulter.

A monotonic transformation of scores can preserve AUC while completely changing the PD levels.

Therefore:

high AUC does not imply calibrated probabilities.

The existing credit-scoring algorithms article covers the broader score-to-probability pipeline.

12. Rare defaults create a seductive low score

If only 1% of cases default, a model that predicts 1% for everyone has a low average squared error.

It may still be useless for identifying which cases are genuinely riskier.

Counterexample: Model A predicts 1% for everyone. Model B separates cases into 0.2%, 1%, 5% and 15% groups but makes a few large mistakes. Model A can look deceptively competitive on the raw Brier score if the sample is dominated by non-defaults.

Falsifier: compare resolution, subgroup calibration and benchmark skill rather than reading BS alone.

13. Inputs and outputs

Inputs can include:

• case-level predicted PD;
• fixed forecast date;
• fixed outcome horizon, such as one year;
• default definition;
• observed default indicator;
• portfolio segment;
• exposure date and censoring flags;
• benchmark forecast;
• calibration-bin rule or smoothing method.

Outputs can include:

• overall Brier score;
• Brier skill score;
• segment-level scores;
• reliability diagram;
• calibration error;
• resolution evidence;
• bootstrap confidence intervals;
• time-series monitoring statistics.

14. Horizon alignment is non-negotiable

A one-year PD must be compared with a one-year outcome window starting from the forecast date.

Mixing six-month outcomes for some cases and eighteen-month outcomes for others changes the event being scored.

Hard boundary: if forecast horizon and outcome horizon do not match, the Brier score no longer verifies the stated probability.

15. Default definition must match the model contract

“Default” can have a regulatory, contractual, accounting or dataset-specific definition.

If the model was trained on one definition but validation labels use another, calibration can appear to deteriorate even when the model is behaving exactly as originally specified.

The event label is part of the algorithm input, not an afterthought.

16. Censoring creates a hidden denominator problem

Suppose a borrower exits the dataset after four months. If the forecast was for one-year default, the full outcome is not observed.

Simply treating the case as a non-default biases the score downward.

For time-to-event settings, survival-analysis extensions such as the integrated Brier score can account for censoring under additional assumptions.

That connects to the survival-analysis default article.

17. Calibration in the large and calibration slope

Beyond grouped reliability plots, a useful diagnostic is to regress outcomes on transformed predicted odds under an appropriate calibration model.

Two broad questions are:

• intercept: is the model systematically too high or too low?
• slope: are predictions too extreme or too compressed?

A model can have an acceptable average Brier score while still showing a material calibration slope problem in the tails.

18. Recalibration is not redevelopment

If ranking remains useful but probability levels drift, a calibration adjustment may be sufficient.

If both ranking and probability levels degrade, the underlying model may need redevelopment.

This distinction is important because changing only the intercept or calibration map preserves more of the original model than rebuilding predictors and coefficients.

The existing model-drift article develops that decision boundary.

19. Regulatory PD calibration is a specific use case

Under internal-ratings frameworks, PD estimation is governed by requirements that go far beyond one scoring rule. The Basel Framework requires ongoing estimation and validation controls, and EBA guidance defines PD calibration in relation to matching assigned PD estimates to long-run average default rates at the relevant calibration level.

The Brier score can support validation, but it is not a substitute for the complete regulatory framework.

The existing PD calibration article owns the long-run grade-to-PD estimation problem.

20. Evidence polarity

Evidence for confidence includes Brier improvement over a relevant benchmark, calibration curves near the identity line within uncertainty, stable scores across time and portfolio segments, useful resolution away from the base rate, and similar conclusions from log loss and calibration diagnostics.

Evidence against confidence includes a good score produced only by a very low default base rate, large subgroup miscalibration, unstable tail bins, score deterioration after a policy or macro regime change, materially different results under small horizon-definition changes, or a model that ranks well but assigns systematically distorted PD levels.

21. Counterexample: the best Brier score is not the best decision rule

Two models can have similar Brier scores but very different consequences at a lending threshold.

If Model A is better calibrated overall but Model B is more accurate near the specific threshold that governs a decision, the operational choice depends on the decision loss function.

Boundary: proper scoring evaluates probabilities; decision analysis evaluates actions under costs and benefits.

22. Counterexample: a model wins because the benchmark is weak

A positive Brier skill score against a naive constant forecast does not prove the model is strong.

It only proves improvement over that chosen reference.

Falsifier: compare against several legitimate baselines, including a previous production model or a simple segment-level empirical forecast.

23. Diagnostics checklist

1. Verify forecast and outcome horizons match.
2. Verify the default definition.
3. Remove duplicate observations.
4. Handle censoring explicitly.
5. Compute the raw Brier score.
6. Compute a benchmark Brier score.
7. Report Brier skill.
8. Plot calibration by probability range.
9. Report uncertainty around observed rates.
10. Measure discrimination separately.
11. Repeat by segment and vintage.
12. Repeat through time.
13. Inspect high-PD and low-PD tails.
14. Compare with log loss.
15. Test whether recalibration fixes the defect.

24. Alternatives and complements

Log loss is also strictly proper and penalises highly confident wrong forecasts more strongly.

Calibration plots show where forecast probabilities deviate from observed frequencies.

AUC/Gini measure discrimination rather than probability accuracy.

Hosmer–Lemeshow-style grouping tests can test calibration but are sensitive to grouping and sample size.

Calibration intercept and slope diagnose systematic level and extremity errors.

Integrated Brier score extends the idea to survival settings with censoring.

25. What would falsify confidence?

Confidence should be withdrawn if the event definition differs from the model target; if outcomes are incompletely observed; if the score is only good because defaults are rare; if calibration fails materially in important segments; if time stability breaks; if a simple benchmark performs equally well; or if the result cannot be reproduced from frozen case-level forecasts and outcomes.

26. Verification and update triggers

Preserve the original case-level PDs before outcomes are known, observation timestamps, horizon rules, default labels, exclusions, benchmark forecasts, segmentation, binning method and score code.

Re-run the audit after material portfolio-mix changes, underwriting-policy changes, macroeconomic regime shifts, default-definition changes, major model recalibration, data-pipeline revisions or sustained Brier deterioration outside its historical control range.

27. Connections to the wider eduKate knowledge estate

The general probability concept is treated separately in How Probability Works. That page owns the broad mathematical idea of calibration across domains.

This Bukit Timah Tutor article is the finance-and-banking projection: it narrows the mechanism to default probabilities, credit-model validation, rare-event baselines and bank-model diagnostics.

The lane is indexed at Finance & Banking Algorithms | Applied Mathematics in Real Financial Systems.

28. Primary and high-quality references

• Glenn W. Brier, Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review, 1950.
• Stefan Siegert, Simplifying and generalising Murphy’s Brier score decomposition, Quarterly Journal of the Royal Meteorological Society, 2017.
• European Banking Authority, Guidelines on the estimation of risk parameters under the IRB Approach.
• Basel Committee on Banking Supervision, IRB approach: minimum requirements to use IRB approach.

Educational boundary: A Brier score evaluates probabilistic forecasts over a defined sample. It does not determine whether a particular borrower should receive credit, what price should be charged, or what regulatory treatment applies.