Quick answer: model drift occurs when the statistical relationship between a model, its inputs and the real world changes enough that yesterday’s performance no longer describes today’s performance. Banks therefore monitor more than headline accuracy. They compare current input distributions with development data, test whether score and outcome relationships remain calibrated, track discrimination and residuals, examine overrides and exceptions, and use outcomes analysis to decide whether a model can continue unchanged, needs recalibration, requires an overlay or should be redeveloped. No single statistic—Population Stability Index included—can prove that a model has or has not drifted.

A model can keep producing numbers perfectly while the meaning of those numbers quietly changes.

Page role: what this article owns

Bukit Timah Tutor already has How Banks Validate Risk Models, which owns the broad model-validation discipline: conceptual soundness, benchmarking, backtesting, challenger models and model-risk governance.

This page owns a narrower production question: after a model has been implemented, how can a bank detect that its inputs, calibration or real-world relationship have changed enough to require intervention?

1. Drift is not one phenomenon

Several different changes are often grouped under “model drift,” but they have different causes and remedies.

Covariate shift: the distribution of model inputs X changes.
Prior-probability shift: the base rate of the target Y changes.
Concept drift: the relationship P(Y|X) changes.
Calibration drift: predicted probabilities no longer match realised frequencies.
Data-pipeline drift: fields, coding, missingness, timing or transformations change even if customer behaviour does not.
Policy drift: the bank changes approvals, limits or interventions, which changes the population later observed.

The same monitoring alarm can therefore have very different meanings. A shifted income distribution may be a real economic change, a new target market or a broken ETL transformation. Diagnosis has to identify the mechanism.

2. Current US supervisory guidance explicitly expects ongoing monitoring

On 17 April 2026, the Federal Reserve, OCC and FDIC issued revised interagency model-risk-management guidance. Federal Reserve SR 26-2 superseded SR 11-7. The revised guidance emphasises a risk-based approach and states that ongoing monitoring should evaluate whether a model continues to perform as expected as products, exposures, activities, clients, data relevance or market conditions change. Persistent deviations outside established performance expectations can warrant adjustment, recalibration or redevelopment.

See Federal Reserve SR 26-2 and the 2026 interagency Supervisory Guidance on Model Risk Management.

The guidance is expected to be most relevant to US banking organisations above the stated size threshold and to other institutions with significant model-risk exposure. It also explicitly excludes generative and agentic AI models from its scope. That exclusion should not be misread as “no governance needed”; it simply means those systems are outside this particular model-risk guidance.

3. Start with the reference population

Suppose a credit model was developed on a reference population with feature distribution P₀(X). In production today, the observed distribution is P_t(X).

The first monitoring question is:

How different is P_t(X) from P₀(X)?

That can be measured feature-by-feature, score-by-score and segment-by-segment. But distribution shift alone does not prove performance deterioration. A variable can move while the model remains well calibrated; conversely, concept drift can damage calibration even when input distributions barely change.

4. Population Stability Index compresses distribution shift into one number

Population Stability Index (PSI) is widely used in credit-risk monitoring because it is easy to compute from binned distributions.

For bins i:

PSI = Σ (A_i − E_i) ln(A_i/E_i)

where E_i is the expected/reference share and A_i the actual/current share.

If the distributions are identical, every term is zero and PSI is zero. As the bin proportions diverge, PSI rises.

5. PSI is useful—but its common thresholds are conventions, not laws

Industry practice often attaches rule-of-thumb bands to PSI, but those bands should not be treated as universal statistical significance thresholds. PSI depends on binning, sample size, zero-cell handling and the shape of the underlying distribution.

Two examples show why:

A huge sample can make a very small distribution change economically visible even when the model remains accurate.
A small but strategically important segment can drift badly while a portfolio-level PSI remains modest because the segment is diluted by the rest of the population.

PSI should therefore trigger investigation, not replace it.

6. Alternative distribution diagnostics answer different questions

Kolmogorov–Smirnov distance: largest gap between two empirical cumulative distributions.
Wasserstein distance: how far probability mass must move to transform one distribution into another.
Jensen–Shannon divergence: a symmetric, bounded information-theoretic comparison built from KL divergence.
Chi-square tests: useful for categorical frequencies under appropriate sample assumptions.
Missingness and category-emergence checks: often detect pipeline changes faster than sophisticated metrics.

A strong monitoring system uses metrics that match the feature type and decision risk instead of forcing every variable through one statistic.

7. Calibration asks whether a predicted probability still means what it says

Suppose a probability-of-default model assigns 10% PD to a group of borrowers. If roughly 10% default over the defined horizon and conditions, the model is locally well calibrated for that group. If 20% default repeatedly, the score may still rank customers correctly while materially understating absolute risk.

Useful calibration diagnostics include:

observed-versus-predicted default rates by score band;
calibration intercept and slope;
Brier score;
log loss;
reliability/calibration plots;
binomial or other outcome-consistency tests where assumptions fit.

Calibration is especially important when a model output feeds pricing, expected loss, capital or policy thresholds where the absolute probability matters—not merely ranking.

8. Discrimination can remain good while calibration decays

Metrics such as AUROC/Gini assess ranking ability: do riskier cases tend to receive higher predicted risk than safer cases? They do not guarantee that 5% means 5%.

A model can preserve nearly the same AUROC after a recession begins while every PD becomes too low because the overall default base rate rises. Conversely, calibration can be corrected with a simple intercept shift while ranking remains unchanged.

This is why one “model accuracy” number is insufficient.

9. Concept drift is the hardest form because P(Y|X) changes

Suppose income, leverage and repayment history used to predict default in one way, but a new economic regime or product structure changes that relationship. The input distribution can look familiar while the conditional outcome relationship changes.

Concept drift therefore requires outcome evidence. Distribution tests on X alone cannot reliably detect it.

Useful signals include:

residuals becoming systematically biased;
calibration curves shifting;
segment-level AUROC/Gini deterioration;
feature-effect instability;
error clustering around new products, channels or economic states;
champion–challenger performance divergence.

10. Outcome delay is a hidden weak link

Many banking outcomes arrive slowly. A 12-month default model cannot be fully validated one week after a portfolio shift. Fraud labels can arrive after disputes. Recovery severity can take years.

This creates a two-speed monitoring problem:

fast indicators: input shift, score shift, missingness, overrides, policy changes;
slow evidence: realised default, loss, cure, recovery or long-horizon performance.

A common failure is to mistake fast proxies for final outcome evidence. They are early warnings, not substitutes.

11. Policy changes can manufacture apparent model drift

Suppose an underwriting model recommends approvals, and the bank tightens its approval policy. The future observed sample now excludes more high-risk applicants. Default performance may improve even if the model itself did not improve.

This is related to the selection-bias problem discussed in How Credit Underwriting Models Learn from Accepted Applicants.

Monitoring therefore needs policy and intervention metadata. Otherwise the bank may attribute a population change to the model when it was caused by the decision process around the model.

12. Recalibration, overlay and redevelopment are different responses

Once deterioration is identified, the remedy should match the mechanism.

Recalibration: adjust probability mapping when ranking remains useful but absolute levels have shifted.
Overlay: apply a temporary governed adjustment when evidence shows a known limitation but redevelopment cannot occur immediately.
Redevelopment: rebuild features, functional form or data when the structural relationship has changed materially.
Usage restriction: narrow the model’s permitted population when it is being applied outside its validated domain.
Retirement: remove the model when it no longer provides reliable value.

The 2026 US interagency guidance explicitly notes that meaningful deterioration can warrant model adjustment, recalibration or redevelopment depending on policy and model risk.

13. A compact monitoring pipeline

Lock the reference data and intended-use population.
Monitor data quality: missingness, ranges, categories, timing and transformation integrity.
Measure feature and score distribution shift.
Track segment composition and policy changes.
Measure discrimination once outcomes become available.
Measure calibration and residual bias.
Review overrides, exceptions and user feedback.
Compare against challenger/benchmark models where useful.
Investigate breached indicators as a mechanism, not merely a threshold.
Decide whether to continue, restrict, recalibrate, overlay, redevelop or retire.
Document the decision and evidence.
Verify subsequent outcomes to see whether the intervention worked.

14. Inputs and outputs

Inputs: development/reference data, current production inputs, model scores, segment labels, policy/override data, realised outcomes, benchmark predictions and model-use metadata.

Outputs: shift metrics, calibration tables, discrimination metrics, data-quality exceptions, monitoring status, root-cause hypotheses, escalation decisions and update/redevelopment triggers.

15. Failure modes

PSI monoculture: one statistic becomes the entire monitoring framework.
Portfolio averaging: severe drift in a small segment is diluted by a stable majority.
Threshold worship: arbitrary bands are treated as universal laws.
Outcome blindness: input distributions are monitored but realised performance is not.
Label delay denial: recent cohorts are declared healthy before outcomes have matured.
Policy contamination: decision-rule changes are mistaken for model changes.
Data-pipeline drift: coding changes create apparent customer behaviour.
Calibration/discrimination confusion: a stable AUROC is treated as proof that probability estimates remain correct.
Silent scope expansion: a model validated for one product or geography is used elsewhere without new evidence.

16. Counterexamples and falsifiers

Claim: “The score distribution is stable, so the model has not drifted.”

Falsifier: observed defaults at every score band have doubled while the score distribution remains almost unchanged. That is calibration or concept drift without major score-distribution drift.

Claim: “PSI is high, so the model is broken.”

Counterexample: the bank intentionally entered a new low-risk customer segment, shifting input distributions while calibration and discrimination remain strong. The population changed, but the model may still be valid within the new domain.

17. Diagnostics

Which features shifted most, and did their data definitions change?
Is score drift explained by one new product, channel or geography?
Did calibration deteriorate before discrimination?
Are errors concentrated around model boundaries or missing-data cases?
Did an approval/collections/pricing policy change before the monitored shift?
How much of performance change remains after controlling for mix?
Does a simple benchmark model deteriorate similarly, suggesting environment rather than model-specific failure?
Are overrides rising because users see a weakness before the statistical dashboard does?

18. Verification and update triggers

recompute shift metrics with alternative binning or distance measures;
verify data lineage before interpreting distribution changes;
wait for sufficiently mature outcomes before declaring long-horizon performance;
recalibrate when ranking remains useful but probability levels have shifted;
redevelop when feature–outcome relationships materially change;
restrict use when production cases move outside the validated population;
review the monitoring suite after product, policy, regulatory or data-source changes;
test whether corrective action actually restores calibration and outcomes.

19. Why monitoring should be risk-based rather than metric-maximalist

A simple low-impact model may not need dozens of daily divergence statistics. A model driving material credit, valuation or capital decisions can justify much richer monitoring. The current 2026 interagency US guidance explicitly emphasises that the nature and rigor of validation and monitoring should align with model approach, use and materiality.

The goal is not to maximise dashboards. It is to detect deterioration early enough to protect the decision that depends on the model.

Connections across the Bukit Timah Tutor finance-algorithms lane

Risk-model validation — the parent discipline; this page owns ongoing drift detection.
Credit scoring — shows why calibration and discrimination are separate properties.
Reject inference and selection bias — explains how policy changes alter the observed population.
Expected credit loss — an example where calibration drift can propagate into financial estimates and overlays.

Research anchors

The deeper lesson

Model drift is a reminder that a predictive model is a relationship between a representation and a changing world. The code may be unchanged, the API may be healthy and the score may still be produced to six decimal places—yet the prediction can lose meaning because customers, markets, products, policies or data changed. Good monitoring therefore reconnects the model to reality through distribution evidence → outcome evidence → diagnosis → intervention → verification.

Educational boundary: This article explains public statistical and banking model-risk concepts. It does not disclose any institution’s monitoring thresholds, internal risk appetite, proprietary models or private control data.

Small Group Tutorials

How Banks Detect Model Drift: Population Stability, Calibration Decay, Distribution Shift, Outcomes Analysis, Recalibration and Redevelopment