How Expected-Shortfall Backtesting Algorithms Test Tail Severity: VaR–ES Joint Forecasts, Exceedance Residuals, Regression Tests and Small-Sample Failure

Reader question: VaR backtesting tells us how often a loss crosses a threshold. How do we test whether the losses beyond that threshold are as severe as an Expected Shortfall model says they should be?

Expected Shortfall backtesting is harder than VaR backtesting because ES is a tail mean, not just a quantile. The algorithm must evaluate both where the tail begins and how deep the realised tail is.

This article owns the tail-severity forecast-testing problem: realised P&L + VaR forecasts + ES forecasts → exceedance residuals, joint calibration tests, regression diagnostics, model-comparison evidence and update triggers.

It does not own the construction of the VaR or ES forecast itself. It also does not replace stress testing. This is public mathematical education, not financial advice.

1. Expected Shortfall answers a different question from VaR

For a continuous loss variable L and confidence level α, VaR is the α-quantile:

VaR_α = q_α(L).

Expected Shortfall averages the tail beyond that quantile:

ES_α = E[L | L ≥ VaR_α]

under the usual continuous-distribution interpretation.

VaR can therefore be backtested with a hit indicator. ES needs information about the magnitudes of the tail losses as well.

2. Why exception counts are insufficient

Suppose two models each produce exactly five VaR exceptions in 500 days. Model A’s exception losses are only slightly beyond VaR. Model B’s exceptions are catastrophically larger.

A VaR exception-count test can treat them similarly. An ES test should not.

This is the core reason ES backtesting requires more than a Bernoulli hit sequence.

3. The simplest tail-severity residual

Let v_t be the VaR forecast and e_t the ES forecast for day t. For losses written as positive numbers, define a tail residual only on VaR-exceedance days. One intuitive object is:

R_t = (L_t − e_t)1{L_t ≥ v_t}.

If the ES forecast is conditionally correct, tail losses should not be systematically above or below the predicted tail mean after the correct conditioning is imposed.

Different published tests use different normalisations and identification functions, but the logic is similar: tail residuals should have the expected sign and mean structure under correct calibration.

4. ES and VaR are statistically linked

Expected Shortfall is defined relative to a quantile. If the VaR forecast is wrong, the set of observations labelled “tail” is wrong too.

This creates a joint-identification problem. Modern statistical theory shows that VaR and ES can be treated jointly with strictly consistent scoring functions even though ES alone does not have the same simple elicitability property as a quantile.

The practical consequence is important: many powerful ES tests evaluate the VaR–ES pair rather than pretending ES is isolated from the threshold that defines its tail.

5. Acerbi–Szekely style backtests

Acerbi and Szekely proposed model-independent backtests designed specifically for ES. Their work demonstrates that ES is backtestable and that the older claim “ES cannot be backtested because it is not elicitable on its own” confuses forecast comparison with hypothesis testing.

Their tests use realised P&L, VaR and/or ES information to examine whether realised tail outcomes are compatible with the ES forecasts.

The important educational point is not one memorised formula. It is that a correct ES model imposes testable restrictions on the average severity of observations selected into the tail.

6. Regression-based ES backtesting

Bayer and Dimitriadis developed regression-based Expected Shortfall tests. The idea is analogous to a forecast-calibration regression: if an ES forecast is correctly specified, suitable intercept and slope restrictions should hold.

A simple conceptual form is:

realised tail outcome = a + b × ES forecast + error.

Correct calibration suggests:

a = 0, b = 1

under the precise regression setup used by the test.

The actual ESR methodology jointly handles quantile and ES structure and uses covariance estimators designed to remain useful under misspecification.

7. Why joint regression is necessary

A quantile forecast can be represented by a check-loss function. Expected Shortfall requires extra structure. Joint VaR–ES regression provides that structure by modelling the threshold and tail mean together.

This lets the test ask whether the ES forecast has systematic bias after accounting for the quantile component.

It also allows multiple test variants: some require auxiliary VaR forecasts, while stricter variants can focus more directly on ES forecast adequacy.

8. Small samples are a severe problem

At a 97.5% confidence level, only 2.5% of observations belong to the tail. In 250 observations, the expected number of tail points is only about 6.25.

At 99%, it is only 2.5.

Estimating the mean of a tail from a handful of observations is noisy. Therefore a failed ES backtest can reflect true model error, sampling noise, or both. A passing test can also have low power.

This is why confidence intervals, bootstrap procedures, rolling windows and multiple diagnostics matter.

9. A worked intuition

Suppose a model reports daily 97.5% VaR of 2 units and ES of 3 units over a period where ten tail losses are observed:

2.1, 2.3, 2.4, 2.5, 2.7, 2.9, 3.0, 3.1, 4.8, 5.2.

The average is 3.10 units. That is close to the ES forecast of 3, but one sample mean alone is not a formal backtest. The tail points are random, the threshold can vary by day and the model’s ES forecast can be time-varying.

A proper test standardises the information so that forecasts from different days can be evaluated together.

10. Inputs and outputs

Inputs can include:

• date;
• realised actual and/or hypothetical P&L;
• VaR forecast;
• ES forecast;
• confidence level;
• portfolio or desk identifier;
• model version;
• market-data version;
• missing-data flags;
• bootstrap or covariance-estimation settings.

Outputs can include:

• VaR exception indicator;
• tail residual;
• tail-loss ratio;
• mean residual test statistic;
• joint VaR–ES calibration statistic;
• ES regression coefficients;
• p-values and confidence intervals;
• rolling-window diagnostics;
• tail-severity attribution by factor or product;
• model-review trigger.

11. Evidence polarity

Evidence for confidence includes tail residuals centred near the theoretical target, stable regression coefficients near their null values, no systematic underprediction during stress, consistent conclusions across adjacent windows and reasonable agreement between actual-P&L and hypothetical-P&L versions after known differences are explained.

Evidence against confidence includes tail losses persistently larger than ES, regression slopes far from one, residuals predictable from volatility or market regimes, results dominated by one or two observations, strong disagreement across reasonable test variants, or systematic failure in exactly the products whose tail risk the model is meant to capture.

12. Counterexample: VaR is wrong but ES looks right

Suppose the ES forecasts happen to match the average of realised large losses, but the VaR threshold is too low and selects far too many observations into the tail.

The model is not jointly calibrated.

Falsifier: backtest the VaR threshold and the ES tail mean together. A correct ES number cannot rescue an incorrect tail definition.

13. Counterexample: VaR passes while ES fails

A model can generate the correct number of VaR exceptions while underestimating their severity.

Falsifier: condition on exceptions and compare realised tail losses with ES. If the mean tail residual is persistently positive, VaR coverage is hiding tail-shape failure.

14. Counterexample: one crisis dominates the result

With a small tail sample, one extreme loss can determine the entire ES backtest.

Falsifier: perform leave-one-out and influence diagnostics. If deleting one observation changes pass to fail or reverses the sign of the estimated bias, report the fragility explicitly.

15. Counterexample: model improves in sample only

Recalibrating after a failed backtest can mechanically improve historical fit.

Falsifier: freeze the recalibrated model and evaluate on subsequent observations. Improvement must survive out-of-sample.

16. Counterexample: nonstationary tail

A tail model calibrated in a low-volatility regime may fail when correlations and volatilities jump.

Falsifier: regress tail residuals on volatility regime indicators. Predictable residual bias means the ES forecast is missing conditional structure.

17. Counterexample: too few observations to distinguish models

Two ES models can have meaningfully different tail predictions yet both pass because the sample contains too few tail observations.

Failure to reject is not proof of equivalence.

Diagnostic: simulate the test’s power under realistic alternatives before interpreting a pass as strong evidence.

18. Basel and regulatory context

The Basel market-risk framework uses Expected Shortfall as a core risk measure under the internal-models approach, while model approval also depends on other tests such as backtesting and P&L attribution. Regulatory implementations can use specific prescribed procedures that are not identical to every academic ES backtest.

Because regulatory text changes over time, use the current Basel Framework when implementing a compliance rule rather than relying on an old textbook summary.

19. Alternatives and complementary tests

VaR backtesting tests quantile coverage and clustering but not tail severity.

Scenario stress testing probes specified extreme states without requiring a precise statistical frequency.

Extreme Value Theory tests and models asymptotic tail shape rather than forecast calibration.

Comparative scoring ranks competing VaR–ES forecasts using strictly consistent joint scores.

Distributional tests evaluate more of the predicted loss distribution than one quantile and one tail mean.

20. Connections to the surrounding Bukit Timah Tutor estate

The natural precursor is VaR backtesting algorithms, which explains coverage and independence tests.

Tail-shape modelling connects to extreme-value tail algorithms.

The regulatory setting connects to FRTB internal-model algorithms.

The full lane is indexed at Finance & Banking Algorithms | Applied Mathematics in Real Financial Systems.

21. What would falsify confidence?

Confidence should be withdrawn if VaR and ES are not jointly calibrated; if tail residuals are systematically biased; if regression restrictions fail; if the result depends on one observation; if residuals remain predictable by regimes or risk factors; if reasonable P&L definitions produce incompatible conclusions; or if the test has too little power for the intended decision.

22. Verification and update triggers

Preserve every VaR forecast, ES forecast, P&L observation, exception flag, residual, model version and test configuration. Revalidate after changes in confidence level, P&L construction, risk-factor mapping, portfolio composition, ES methodology, regulatory requirements or statistical test implementation.

23. Primary and high-quality references

• Carlo Acerbi and Balazs Szekely, Backtesting Expected Shortfall, MSCI Research Insight, 2014.
• Sebastian Bayer and Timo Dimitriadis, Regression-Based Expected Shortfall Backtesting, Journal of Financial Econometrics.
• Timo Dimitriadis and Sebastian Bayer, A Joint Quantile and Expected Shortfall Regression Framework.
• Basel Committee on Banking Supervision, Basel Framework, current source for market-risk requirements.

Educational boundary: An ES backtest measures whether forecast tail severity is compatible with observed outcomes under a chosen statistical test. It does not prove the future tail is known, and it is not a trading signal.