How ROC/AUC, CAP/Accuracy-Ratio and KS Algorithms Test Credit-Model Discrimination: Ranking Defaults, Thresholds, Gini Equivalence, Sampling Uncertainty and Validation Failure

Reader question: A credit model can output probabilities or scores, but before asking whether those probabilities are numerically calibrated, how do we test whether the model can actually rank riskier borrowers ahead of safer borrowers?

Credit-model discrimination metrics answer that ranking question. ROC curves, Area Under the Curve (AUC), Cumulative Accuracy Profiles (CAP), Accuracy Ratio/Gini and the Kolmogorov–Smirnov (KS) statistic all compare the score distributions of defaulted and non-defaulted observations, but they summarise different parts of that separation.

This article owns one narrow public-learning job: testing the discriminatory power of a binary credit-risk score. It does not own PD calibration, Brier scoring, scorecard construction, reject inference or regulatory capital. Those are adjacent roles already represented elsewhere in the Bukit Timah Tutor finance-and-banking-algorithms estate.

This is public mathematical and computational education. It is not a lending decision rule, a regulatory approval opinion or personalised financial advice.

1. Discrimination is not calibration

Suppose two borrowers receive predicted default probabilities of 4% and 8%.

If the 8% borrower is genuinely riskier, the model has ranked this pair correctly. That contributes to discrimination.

But if actual comparable default frequencies are 1% and 2%, the probability levels are poorly calibrated even though the ranking is correct.

Therefore:

discrimination asks who is riskier;

calibration asks whether the numerical probabilities are right.

A model can do well on one and poorly on the other.

2. Set the score direction first

Some credit systems assign higher score = safer. Others assign higher score = riskier.

If this convention is reversed accidentally, AUC can appear below 0.5 even when the model is highly informative.

Every discrimination engine should store an explicit score direction and event label before any metric is computed.

3. Threshold classification

Let y = 1 denote default and let s be a risk score where higher means riskier.

For threshold c, classify:

predicted default if s ≥ c.

This creates:

• true positives (TP): defaulted and flagged;
• false positives (FP): non-defaulted but flagged;
• true negatives (TN): non-defaulted and not flagged;
• false negatives (FN): defaulted but not flagged.

From these:

TPR = TP/(TP+FN)

and:

FPR = FP/(FP+TN).

Changing c changes both rates.

4. The ROC curve

The Receiver Operating Characteristic curve plots:

FPR on the horizontal axis

against:

TPR on the vertical axis

as the decision threshold moves through the score distribution.

A random ranking lies near the diagonal:

TPR = FPR.

A perfect ranking reaches the upper-left corner: all defaults are identified before any non-default is incorrectly flagged.

5. AUC as a pairwise probability

For continuous scores with the chosen direction, AUC has a powerful interpretation:

AUC = P(score of a randomly chosen defaulter > score of a randomly chosen non-defaulter)

with half-credit conventionally assigned to ties.

This links AUC to the Mann–Whitney/Wilcoxon rank statistic.

Examples:

• AUC = 0.50 → random ranking;
• AUC = 1.00 → perfect ranking;
• AUC = 0.75 → a random default/non-default pair is ordered correctly about 75% of the time under the standard interpretation.

AUC is therefore threshold-free in the sense that it aggregates ranking performance across all possible score cut-offs.

6. AUC does not tell us calibration

If every predicted PD is multiplied by two, the ranking may remain identical.

ROC and AUC can therefore remain unchanged even though probability calibration becomes much worse.

This is why Brier-score algorithms and PD calibration algorithms remain separate validation owners.

7. The Cumulative Accuracy Profile

The CAP curve sorts the portfolio from highest predicted risk to lowest and asks how rapidly the cumulative proportion of realised defaults is captured.

Schematically:

• x-axis: cumulative fraction of all observations, ordered by score;
• y-axis: cumulative fraction of all defaults captured.

A perfect model captures all defaults within the riskiest fraction of the portfolio equal to the overall default rate.

A random model follows the diagonal.

8. CAP depends visually on the base default rate

Unlike ROC, the shape of the CAP curve depends on how many defaults exist in the sample.

Two portfolios with identical ranking quality but different default rates can have visually different CAP shapes.

The Basel validation literature explicitly warns that direct visual comparison of CAP curves across different portfolio compositions can therefore be misleading.

9. Accuracy Ratio / Gini

The Accuracy Ratio (AR), often called the Gini coefficient in credit scoring, compares the area between the model CAP and the random CAP with the area between the perfect CAP and random CAP.

Under standard binary-classification conventions, AR and ROC-AUC are linearly related:

AR = 2 × AUC − 1.

So:

• AUC = 0.50 → AR = 0;
• AUC = 0.75 → AR = 0.50;
• AUC = 1.00 → AR = 1.

Different software or institutions can use slightly different Gini/AR definitions, so the exact convention should be recorded.

10. ROC/AUC and AR contain closely related ranking information

If the same sample, labels, tie treatment and score direction are used, AUC and AR should reconcile through their transformation.

Diagnostic: if reported AUC is 0.80 but reported AR is 0.30, either the metric conventions differ, the samples differ, or there is an implementation error.

11. The KS statistic

The two-sample Kolmogorov–Smirnov statistic examines the maximum separation between the cumulative score distributions of defaulters and non-defaulters.

Let:

F_D(s) = empirical CDF of scores for defaults,

F_N(s) = empirical CDF for non-defaults.

Then:

KS = sup_s |F_D(s) − F_N(s)|.

Credit-scoring implementations often report the score threshold where this maximum occurs.

KS therefore highlights the single point of maximum distributional separation, whereas AUC averages ranking performance across the full range.

12. A worked miniature example

Suppose five borrowers defaulted with risk scores:

95, 80, 75, 60, 55

and five did not default:

70, 50, 40, 30, 20.

Most default/non-default pairs are correctly ordered because most default scores exceed most non-default scores, so AUC will be well above 0.5.

At a threshold around 55–70, the empirical default and non-default cumulative distributions may be farthest apart, generating the KS maximum.

The CAP will show defaults accumulating rapidly among the first high-risk-ranked observations.

All three views describe the same score separation from different angles.

13. Inputs and outputs

Inputs can include:

• model score or predicted PD;
• binary realised default outcome;
• event horizon and default definition;
• score direction;
• observation weights if justified;
• sample dates;
• segment identifiers;
• development/validation flag;
• treatment of ties and censored observations.

Outputs can include:

• ROC curve;
• AUC;
• AUC confidence interval;
• CAP curve;
• Accuracy Ratio / Gini;
• KS statistic;
• KS threshold;
• segment-level metrics;
• bootstrap distributions;
• development-versus-validation deltas;
• drift and degradation flags.

14. Confidence intervals matter

An AUC estimate of 0.72 from 10,000 observations with 1,000 defaults is much more precisely estimated than an AUC of 0.72 from 300 observations with six defaults.

Useful uncertainty methods include:

• DeLong-type asymptotic intervals for AUC;
• bootstrap intervals;
• cluster/bootstrap methods when observations are dependent;
• exact or permutation methods in specialised settings.

A point metric without uncertainty can create false precision, especially in low-default portfolios.

15. Low-default portfolios are difficult

Discrimination is identified by comparisons involving actual defaults. If only five defaults occur, there are very few event observations from which to infer ranking quality.

A single additional default can move AUC, KS and CAP materially.

Falsifier: bootstrap the default sample. If the ranking metrics vary widely, the correct conclusion is high uncertainty, not a stable performance grade.

16. Ties matter

Rating systems often assign many borrowers to the same grade. This creates score ties.

AUC conventionally gives a tied default/non-default pair half credit. CAP and KS also need deterministic treatment of repeated score values.

Changing from continuous scores to coarse grades can lower observed AUC even when no pair is reversed, because more pairs become ties.

This is conceptually similar to the tie compression created by isotonic probability calibration.

17. Evidence polarity

Evidence for confidence includes:

• AUC/AR substantially above random with narrow uncertainty;
• ROC/CAP shapes broadly concave in the expected direction;
• KS separation is stable across adjacent samples;
• development and validation metrics are similar;
• performance persists across meaningful segments;
• AUC, AR and pairwise ranking calculations reconcile;
• metric deterioration is explainable by population changes rather than implementation errors.

Evidence against confidence includes:

• AUC near 0.5;
• very wide confidence intervals;
• large validation drop from development;
• ROC/CAP non-concavity or score reversals;
• KS threshold moving wildly over time;
• metrics strong only in one segment;
• AUC and AR failing their expected reconciliation;
• performance disappearing after leakage controls.

18. Counterexample: high AUC, terrible calibration

Suppose a model predicts 20%, 40%, 60%, 80% default probabilities for increasingly risky groups, but the true long-run rates are 2%, 4%, 6%, 8%.

The ranking can be almost perfect, so AUC can be excellent. Yet every probability is grossly too high.

Falsifier: pair discrimination tests with calibration metrics and reliability analysis. High AUC is not evidence that PD levels are correct.

19. Counterexample: stable AUC, worsening decisions

AUC weights all thresholds according to rank comparisons, not the institution’s operational cost function.

Suppose a bank can review only the riskiest 1% of applications. The model may maintain AUC while becoming much worse specifically in that extreme top-risk region.

Falsifier: add precision/recall, lift, partial AUC or top-decile capture diagnostics at the operating region.

20. Counterexample: CAP comparison across different default rates

Portfolio A has a 1% default rate; Portfolio B has a 10% default rate. Their CAP curves can have different shapes even with similar discriminatory power.

Falsifier: do not infer relative discrimination from visual CAP curvature alone. Compare AR/AUC with uncertainty and use common sample definitions.

21. Counterexample: KS overfits one threshold

KS reports the maximum empirical separation. In a small sample, the maximum can occur at a threshold created by random fluctuation.

Falsifier: bootstrap the KS statistic and threshold. A stable model should not rely on one fragile score cut-off.

22. Counterexample: policy changes contaminate the sample

A lending model is used to approve only high-score applicants. Future observed defaults then come mainly from the approved region.

The validation sample is no longer a random slice of the original applicant population.

Discrimination metrics can change because of selection, not because the score function changed.

Falsifier: document acceptance policy and sample selection. See reject-inference and selection-bias algorithms for the missing-label problem.

23. Counterexample: data leakage creates spectacular AUC

A feature accidentally includes information recorded after default—for example, a collection-status code.

The development AUC can become nearly perfect.

Falsifier: rebuild every feature with strict prediction-time timestamps. A large AUC collapse after leakage removal is evidence that the original discrimination was artificial.

24. ROC shape diagnostics

A properly oriented score often generates a broadly concave ROC curve. Strong local reversals can indicate that some score regions are misordered.

Basel validation research notes that non-concavity can signal suboptimal use of information in the score specification.

This does not mean every empirical wiggle is a model failure; small samples naturally create jagged curves. Smoothing should not hide real reversals without investigation.

25. CAP/ROC relation to monotone PDs

If higher risk scores correspond monotonically to higher conditional default probabilities, the ordering structure supports concave ROC/CAP behaviour.

Persistent non-concavity can therefore be a clue that:

• score direction is wrong;
• some bins are misordered;
• the model omitted interaction/nonlinearity;
• segments should not be pooled.

Score construction tools such as Weight-of-Evidence binning can impose or encourage monotone score relationships, but discrimination still requires independent validation.

26. Segment diagnostics

A global AUC can hide local weakness.

Useful segment checks include:

• product;
• region;
• origination channel;
• secured/unsecured;
• rating band;
• time vintage;
• new versus existing customers.

Metrics should not be compared mechanically across tiny segments. Report sample sizes and default counts beside every metric.

27. Time stability

Discrimination can deteriorate while calibration is separately adjusted.

Track rolling or vintage AUC/AR/KS together with:

• population stability;
• default-rate changes;
• score-distribution shift;
• policy changes;
• feature missingness.

See model-drift algorithms for the broader monitoring framework.

28. Alternatives and complements

Precision–recall curves: useful when defaults are rare and operational focus is on positive predictions.

Lift/capture rate: measures how many defaults are concentrated in the top-ranked fraction.

Partial AUC: focuses on a restricted false-positive operating range.

Brier/log loss: evaluate probability accuracy, not only ranking.

Calibration plots: compare predicted and realised event frequencies.

Decision-cost metrics: evaluate performance at actual operational thresholds.

29. Weak links

• wrong event label;
• score direction reversed;
• post-event leakage;
• selection/reject bias;
• tiny default counts;
• inconsistent tie handling;
• development and validation samples overlap;
• CAPs compared across different base rates without caution;
• KS threshold treated as universally optimal;
• one global metric hiding segment failure.

30. What would falsify confidence?

Confidence should be withdrawn if out-of-sample AUC approaches random; if AUC/AR uncertainty is too wide for the intended decision; if ROC/CAP curves show persistent reversals; if KS is unstable; if metrics collapse after leakage/selection controls; if important segments have materially weaker ranking; or if the model’s operational region performs poorly despite a respectable global AUC.

31. Verification and update triggers

Preserve model version, score direction, sample definition, outcome horizon, default definition, tie policy, weighting, development/validation split, AUC/AR/KS code and confidence-interval method.

Revalidate when:

• score model changes;
• credit policy changes;
• portfolio composition changes;
• default definition changes;
• default counts become sparse;
• selection mechanisms change;
• rolling discrimination deteriorates;
• new products/segments are added.

32. Primary and high-quality references

• Basel Committee on Banking Supervision, Studies on the Validation of Internal Rating Systems, Working Paper No. 14, 2005. Discusses CAP, Accuracy Ratio, ROC/AUC and other discriminatory-power measures.
• Basel Committee on Banking Supervision, publication page for Studies on the Validation of Internal Rating Systems.
• James A. Hanley and Barbara J. McNeil, The Meaning and Use of the Area under a Receiver Operating Characteristic Curve, for the pairwise-probability interpretation of AUC.
• Elizabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson, Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves, Biometrics.
• Federal Reserve, Supervisory Guidance on Model Risk Management, for validation and ongoing monitoring principles.

Educational boundary: Discrimination metrics evaluate ranking. They do not by themselves tell whether PDs are calibrated, whether a lending policy is fair or optimal, or whether a regulatory model meets all supervisory requirements.