How Isotonic-Regression Calibration Algorithms Turn Credit Scores into Monotone Probabilities: PAVA, Stepwise Mapping, Brier Loss, Ranking Ties and Drift Diagnostics

Reader question: A credit model may rank borrowers correctly while its numerical probabilities are systematically too high, too low or distorted. Can we repair the probabilities without rebuilding the ranking model?

Isotonic regression provides one answer. It learns a non-parametric, non-decreasing mapping from a model score to an observed event frequency. The algorithm does not assume a logistic S-curve. Instead, it preserves ordering while pooling adjacent score regions whenever their empirical event rates violate monotonicity.

This article owns a narrow computational job: post-model monotone probability calibration. Its input is an existing score or probability ranking plus labelled calibration outcomes. Its output is a stepwise monotone probability map. It does not own regulatory long-run PD estimation, rating-grade design, scorecard construction or discrimination testing. Those are adjacent jobs already covered elsewhere in the Bukit Timah Tutor finance-and-banking-algorithms estate.

This is public mathematical and computational education. It is not personalised financial advice, a lending decision rule, or a claim that one calibration method is appropriate for every regulatory or business use.

1. Ranking and calibration are different questions

Suppose Model A assigns scores to borrowers so that riskier borrowers usually receive higher scores. That is a discrimination or ranking property.

Now suppose the model says a group has a 4% default probability but 7% actually default over repeated comparable samples. That is a calibration problem.

A model can rank well and calibrate poorly. Conversely, a crude model can be approximately calibrated on average while ranking borrowers badly.

This distinction is why a post-processing calibration layer can be mathematically legitimate: it can change probability levels while trying to preserve the underlying ordering information.

2. The isotonic optimisation problem

Let the original model produce scores:

s₁, s₂, …, s_n.

Let outcomes be:

y_i ∈ {0,1},

where 1 denotes the event of interest, such as default within a specified horizon.

After sorting observations so that:

s₁ ≤ s₂ ≤ … ≤ s_n,

isotonic least-squares calibration solves:

min Σ w_i(y_i − p_i)²

subject to:

p₁ ≤ p₂ ≤ … ≤ p_n.

The weights w_i can be one, exposure weights, sampling weights, or another justified scheme. The fitted p_i values become calibrated probabilities.

For binary outcomes, minimising squared error corresponds to minimising in-sample Brier loss under the monotonicity constraint.

3. Why the answer becomes stepwise

Imagine three adjacent score bands with empirical event rates:

2%, 6%, 4%.

If higher score means higher risk, the final two bands violate monotonicity because 4% follows 6%.

Isotonic regression does not force a predetermined functional curve through these points. It pools the violating neighbours. If the bands have equal weight, the pooled rate becomes:

(6% + 4%)/2 = 5%.

The resulting sequence is:

2%, 5%, 5%.

If a pooled block still violates a neighbouring block, pooling continues. This produces flat steps wherever the data do not support finer monotone distinctions.

4. Pool Adjacent Violators Algorithm (PAVA)

The standard computational engine is the Pool Adjacent Violators Algorithm.

1. Sort observations or pre-aggregated score groups in score order.
2. Start each observation/group as its own block.
3. Compute each block’s weighted mean outcome.
4. Scan for adjacent blocks whose fitted means violate the required order.
5. Merge the violating blocks.
6. Replace them with their combined weighted mean.
7. Repeat until all adjacent block means are monotone.

The mechanism is deceptively simple. Its strength comes from solving a globally constrained optimisation problem through local pooling.

5. A worked example

Suppose a score model produces five ordered groups with 100 observations each and default counts:

• Group 1: 1 default → 1%
• Group 2: 4 defaults → 4%
• Group 3: 9 defaults → 9%
• Group 4: 6 defaults → 6%
• Group 5: 15 defaults → 15%

The sequence 1%, 4%, 9%, 6%, 15% violates monotonicity at Groups 3 and 4.

PAVA pools Groups 3 and 4:

(9 + 6)/(100 + 100) = 7.5%.

The final fitted sequence becomes:

1%, 4%, 7.5%, 7.5%, 15%.

The algorithm has refused to claim that Groups 3 and 4 have reliably different probability levels given the observed sample.

6. What isotonic regression assumes

The central structural assumption is not logisticity or normality. It is monotonicity:

higher model score should not imply lower true event probability.

That is weaker than assuming a particular parametric calibration curve, but it is still a substantive assumption.

If the underlying model score has a genuinely non-monotonic relationship with risk, isotonic regression cannot repair that structure faithfully. It will impose monotonicity whether the world supports it or not.

7. Calibration data must be out-of-sample relative to model fitting

A common implementation failure is to fit the base model and calibrator on the same observations, then report the resulting probability fit as if it were out-of-sample evidence.

A safer workflow separates:

• training data for the base scoring model;
• calibration data for the isotonic map;
• validation/test data for evaluating the combined system.

Cross-validation or cross-fitting can help use limited data efficiently while preserving this separation.

8. Inputs and outputs

Inputs can include:

• base-model score or uncalibrated probability;
• binary realised outcome;
• event horizon and outcome definition;
• observation or exposure weight;
• calibration sample dates;
• segment identifiers;
• cross-validation folds;
• minimum-block-size or clipping policy if used.

Outputs can include:

• monotone score-to-probability map;
• PAVA block boundaries;
• block event rates and weights;
• out-of-sample Brier score;
• log-loss diagnostics;
• reliability diagram;
• number and size of tied probability blocks;
• AUC before and after calibration;
• bootstrap uncertainty;
• drift and recalibration triggers.

9. Why isotonic calibration can alter AUC slightly

A strictly increasing transformation preserves every pairwise ranking, so ranking metrics such as ROC-AUC remain unchanged.

Isotonic regression is only non-decreasing. Its flat steps create ties. Two borrowers who previously had different scores can receive exactly the same calibrated probability.

Those new ties can change AUC slightly even though the calibrator has not reversed their ordering.

This is an important diagnostic distinction: a small AUC change after isotonic calibration may be caused by tie compression rather than a new ranking model.

10. Small samples are the main statistical danger

Because isotonic regression is non-parametric, it has more flexibility than a one- or two-parameter calibration curve. That flexibility needs data.

With few events, a sequence of chance outcomes can create unstable steps. The fitted curve can contain probabilities near 0 or 1 simply because small blocks happened to contain no events or only events.

Modern calibration guidance and empirical studies therefore emphasise that isotonic regression is more prone to overfitting than sigmoid-style calibration when calibration samples are small.

11. Rare-default portfolios make this harder

In low-default portfolios, the number of non-default observations may be large while the number of defaults is tiny. Calibration information is driven primarily by the events.

For example, 10,000 observations with only eight defaults do not provide the same calibration information as 10,000 observations with 800 defaults.

A PAVA curve can still be computed, but mathematical computability is not evidence of statistical stability.

In these settings, credible intervals, pooling across justified horizons/segments, Bayesian methods, parametric calibration, or explicit conservatism may be preferable depending on the intended use.

12. Weighted isotonic regression

If each calibration row represents a different number of underlying observations, or observations arise from sampling schemes, weights matter.

When two violating blocks A and B are pooled, the new fitted value is:

p_A∪B = (W_Ap_A + W_Bp_B)/(W_A + W_B).

Ignoring legitimate weights can change block boundaries and fitted probabilities.

13. Reliability diagrams: binning can mislead

Traditional reliability diagrams often place forecasts into arbitrary bins, then plot mean forecast probability against event frequency in each bin.

The picture can change when the analyst changes the bin count or boundaries.

Research on stable reliability diagrams uses isotonic regression and PAVA to produce data-driven monotone reliability curves, avoiding some arbitrary-binning problems and supporting uncertainty analysis.

This is useful because the calibrator itself and the diagnostic chart can be based on closely related order-restricted mathematics.

14. Evidence polarity

Evidence for using isotonic calibration includes:

• the base score has credible monotone risk ordering;
• the calibration sample contains enough events;
• out-of-sample Brier score improves;
• reliability curves become closer to the diagonal;
• the number and size of PAVA blocks are stable across bootstrap samples;
• probabilities remain stable across adjacent validation windows;
• ranking performance is not materially damaged;
• alternative calibrators do not produce materially superior out-of-sample results.

Evidence against confidence includes:

• large probability jumps caused by one or two events;
• many tiny blocks;
• extreme 0/1 probabilities in sparse tails;
• strong differences between folds;
• better training Brier score but worse test Brier score;
• non-monotone empirical reliability that persists systematically;
• material performance decay after population shift.

15. Counterexample: perfect in-sample calibration, poor future calibration

Because isotonic regression minimises squared calibration error under monotonicity on the calibration sample, it can fit that sample very well.

Suppose a small sample contains an unusually high cluster of defaults in one narrow score region. PAVA may create a sharp step around that cluster.

On a new sample the cluster disappears, but the sharp step remains and probabilities are now distorted.

Falsifier: evaluate the calibration map on a truly held-out period and bootstrap the step boundaries.

16. Counterexample: the score itself is wrong

Suppose borrowers with middle scores are actually riskier than borrowers with high scores because the model omitted an important nonlinear feature.

Isotonic regression must preserve score ordering, so it cannot assign the middle group a higher probability than the high group if the scores say otherwise.

Falsifier: inspect calibration conditional on meaningful subgroups and test discrimination independently. Persistent non-monotone outcome patterns indicate a ranking-model problem, not merely a calibration problem.

17. Counterexample: population drift

A calibrator fitted during a benign credit period may map a score of 700 to 1% default probability. After an economic regime change, the same score distribution can experience materially higher default rates.

The monotone shape may remain sensible while the probability levels become stale.

Falsifier: monitor realised-versus-predicted frequencies by time and score region. Recalibration is triggered by level drift even when ranking metrics remain stable.

18. Counterexample: segment mixing

If two portfolios have different base rates or risk mechanisms, pooling them can create a calibration curve that is correct for neither.

For example, secured mortgages and unsecured revolving credit can have different default processes even if their model scores share a numeric range.

Falsifier: compare reliability by segment. If the global curve hides systematic opposite errors, separate calibration segments or a richer model may be required.

19. Counterexample: extrapolation outside the observed score range

Isotonic regression learns from observed score locations. At the extremes, new scores may fall outside the calibration range.

A software library may clip to the nearest fitted probability. That is an implementation convention, not evidence about the true tail.

Falsifier: report the proportion of production scores outside the calibrated range and treat material extrapolation as a model-use warning.

20. Alternatives

Platt/sigmoid scaling uses a low-dimensional logistic mapping. It is smoother and less flexible, which can help in small samples but can miss non-sigmoid distortions.

Beta calibration uses a richer parametric family that includes the identity mapping and can correct distortions that logistic calibration handles poorly.

Temperature scaling is common for neural-network confidence calibration but is narrower than isotonic calibration.

Bayesian or hierarchical calibration can explicitly represent parameter uncertainty and partial pooling in low-event settings.

Rebuilding the base model is necessary when the score ordering itself is structurally wrong.

21. How this differs from regulatory long-run PD calibration

The Bukit Timah Tutor article How Probability-of-Default Calibration Algorithms Turn Rating Grades into Long-Run One-Year PDs owns the broader regulatory/statistical problem of mapping grades to long-run one-year default probabilities.

This isotonic page owns only one narrower tool: how a monotone non-parametric mapping is computed and validated. An institution using an IRB framework still has to satisfy the relevant definitions of calibration sample, long-run average default rate, representativeness, margin of conservatism and governance. PAVA does not replace those requirements.

22. Connections to the surrounding knowledge estate

Weight-of-Evidence binning owns scorecard feature/score construction and monotonic binning, not post-hoc probability calibration.

Brier-score algorithms own a proper scoring-rule audit of probability forecasts; Brier loss is also the natural squared-error objective underlying ordinary isotonic calibration.

Conformal prediction answers a different question: set/coverage uncertainty under exchangeability-type assumptions rather than point-probability recalibration.

The broader lane is indexed at Finance & Banking Algorithms | Applied Mathematics in Real Financial Systems.

23. Weak links

• Score-direction error: higher scores accidentally mean lower risk in one system and higher risk in another.
• Outcome leakage: calibration labels contain information unavailable at prediction time.
• Event-definition drift: “default” changes between calibration and production.
• Time-window leakage: calibration and validation periods overlap.
• Too few events: block estimates become unstable.
• Weight misuse: sample weights are omitted or double-counted.
• Segment pooling: different risk mechanisms are forced into one curve.
• Extremes clipping: out-of-range production scores are silently assigned endpoint probabilities.

24. What would falsify confidence?

Confidence should be withdrawn if held-out Brier/log-loss deteriorates; if bootstrap PAVA curves vary materially; if score regions display persistent non-monotone outcomes; if calibration depends strongly on arbitrary sample boundaries; if production scores move outside the calibrated support; if base rates or segment composition change materially; or if a simpler calibrator performs as well with substantially greater stability.

25. Verification and update triggers

Preserve the base-model version, score direction, calibration sample, outcome definition, event horizon, weights, fold assignments, PAVA blocks, endpoint policy and all held-out metrics.

Re-run validation when:

• the base model changes;
• score distribution shifts;
• default/event rates change materially;
• portfolio mix changes;
• new segments are introduced;
• data definitions change;
• calibration blocks become sparsely populated;
• reliability diagnostics deteriorate.

26. Primary and high-quality references

• Alexandru Niculescu-Mizil and Rich Caruana, Predicting Good Probabilities with Supervised Learning, ICML 2005. Compares isotonic regression and Platt scaling and studies calibration-sample effects.
• Bianca Zadrozny and Charles Elkan, Transforming Classifier Scores into Accurate Multiclass Probability Estimates, KDD 2002.
• Timo Dimitriadis, Tilmann Gneiting and Alexander I. Jordan, Stable Reliability Diagrams for Probabilistic Classifiers, PNAS, 2021.
• Meelis Kull, Telmo Silva Filho and Peter Flach, Beta Calibration: A Well-Founded and Easily Implemented Improvement on Logistic Calibration, AISTATS 2017.
• European Banking Authority, Guidelines on PD Estimation, LGD Estimation and Treatment of Defaulted Exposures, for the broader regulatory meaning of PD calibration.
• Federal Reserve, Supervisory Guidance on Model Risk Management, for validation, monitoring and model limitations.

Educational boundary: Isotonic regression is a calibration operator, not a lending policy and not proof that the underlying score model is correct. A monotone map can repair probability levels only within the information and assumptions supplied to it.