How Diebold–Mariano Algorithms Compare Financial Forecasts: Loss Differentials, HAC Variance, Overlapping Horizons, Small-Sample Corrections and Nested-Model Failure

Reader question: Two financial forecasting models produce different errors. One has a lower average loss—but is that difference large enough to treat as evidence of genuinely better predictive accuracy rather than sampling noise?

The Diebold–Mariano (DM) framework turns this into a statistical test. Instead of comparing two headline RMSE numbers, it constructs a time series of loss differentials, estimates the long-run variance of their mean, and tests whether expected relative loss is zero.

This article owns one exact computational job: out-of-sample equal-predictive-accuracy testing for two forecasts under a chosen loss function. It does not own forecast construction, model calibration, general backtesting, conditional predictive-ability testing in full, or multiple-model data-snooping correction.

This is public mathematical and computational education. It is not a recommendation to trade on whichever model wins a forecast test, and statistical significance does not establish economic value.

1. Start with the forecast target

Let y_t be the realised quantity and let two forecasts be:

ŷ_1,t|t-h, ŷ_2,t|t-h.

The forecast horizon is h.

Errors are:

e_1,t = y_t−ŷ_1,t|t-h,

e_2,t = y_t−ŷ_2,t|t-h.

The DM test does not decide what target should be forecast. The target, horizon and information set must already be defined correctly.

2. Loss functions encode the comparison

Choose a loss function L(e).

Common examples include:

Squared error: L(e)=e².

Absolute error: L(e)=|e|.

For asymmetric decisions, loss can penalise positive and negative errors differently.

The original Diebold–Mariano framework is valuable partly because loss need not be quadratic or symmetric.

3. The loss differential

Define:

d_t = L(e_1,t) − L(e_2,t).

Interpretation:

• d_t>0 → model 2 had lower loss at t;
• d_t<0 → model 1 had lower loss;
• d_t=0 → equal realised loss.

The sample mean is:

d̄ = (1/T)Σd_t.

The null hypothesis is:

H₀: E[d_t] = 0.

4. Why comparing only average RMSE is incomplete

Suppose model A has RMSE 1.00 and model B has 0.98.

That difference may be stable across 10,000 observations—or driven by three unusual dates in a sample of 60.

A forecast comparison needs the sampling uncertainty of the loss difference, not only the average difference.

5. The basic DM statistic

The test statistic is:

DM = d̄ / √(V̂(d̄)).

The central problem is estimating:

V(d̄).

If d_t were independent, this would be simple. Financial forecast-loss differentials are often serially correlated and heteroskedastic, so a long-run variance estimate is required.

6. Long-run variance

Let γ_k be autocovariance of d_t at lag k.

The long-run variance is:

Ω = γ₀ + 2Σ_k=1^∞γ_k.

Then:

V(d̄) ≈ Ω/T.

In practice Ω is estimated with a HAC/Newey–West-type estimator or a horizon-motivated truncated covariance estimator.

7. Why overlapping multi-step forecasts create dependence

Suppose every day we forecast a five-day return.

Forecast errors ending on adjacent days share four days of realised returns.

The resulting loss differentials are mechanically serially dependent even if the underlying one-day innovations are independent.

Ignoring this overlap understates the variance of d̄ and can create false significance.

8. Horizon-based autocovariance truncation

Under some h-step forecast structures, a common estimator includes autocovariances through lag h−1:

Ω̂ = γ̂₀ + 2Σ_k=1^h−1γ̂_k.

This should not be used mechanically when the loss differential has additional serial dependence beyond horizon overlap.

Falsifier: inspect d_t autocorrelation and compare with broader HAC bandwidth choices.

9. HAC estimation is part of the test

Using a Bartlett/Newey–West kernel, a generic estimator is:

Ω̂ = γ̂₀ + 2Σ_k=1^L(1−k/(L+1))γ̂_k.

The bandwidth L affects the standard error.

A result that flips significance across modest defensible bandwidths is not a robust forecast-comparison conclusion.

10. Small-sample correction

Harvey, Leybourne and Newbold proposed a finite-sample correction frequently applied to DM statistics:

DM* = DM × √[(T+1−2h+h(h−1)/T)/T].

The corrected statistic is often compared with a Student-t reference distribution with T−1 degrees of freedom.

This correction can improve finite-sample size properties, especially when T is not large relative to h.

11. The sign convention must be documented

With:

d=L₁−L₂,

positive d̄ favours model 2.

If software defines the differential in the reverse order, the sign reverses.

Falsifier: manually calculate d̄ from a tiny example before interpreting any positive/negative statistic.

12. One-sided versus two-sided alternatives

A two-sided test asks:

E[d] ≠ 0?

A one-sided test might ask:

E[d] > 0?

if the predefined hypothesis is that model 2 is superior.

Choosing the direction after observing the sign doubles the opportunity for selective inference.

13. The loss function changes the scientific question

Model A can beat model B under squared error but lose under absolute error.

That is not a contradiction.

Squared error emphasizes large misses; absolute error gives linear penalty.

A risk-tail forecast may require a quantile/check loss or another proper scoring rule rather than generic MSE.

14. Quantile forecasts

For α-quantile forecasts q_t, the pinball/check loss is:

L_α(y,q) = (α−1{y<q})(y−q).

DM can compare two quantile forecasts using the difference in check losses.

This is a distinct job from testing whether a VaR model has correct exception coverage.

CAViaR algorithms use check loss to estimate dynamic quantiles; DM can compare out-of-sample quantile losses from competing forecast systems.

15. Expected shortfall needs suitable scoring structure

Expected shortfall is not elicitable alone by a standard single-dimensional strictly consistent score, but VaR and ES are jointly elicitable under appropriate scoring functions.

Comparisons of VaR–ES forecast systems should therefore use loss/scoring functions aligned with the joint target rather than arbitrary squared errors on ES alone.

See expected-shortfall backtesting algorithms.

16. Nested models are a major failure case

Suppose model 1 is a random walk and model 2 is the same model plus an extra predictor whose true coefficient is zero.

The models are nested.

Under the null, the larger model estimates an unnecessary coefficient, adding forecast noise. Standard DM asymptotics for non-nested equal-accuracy comparisons can become inappropriate.

Clark–McCracken and Clark–West methods were developed for nested forecast comparisons.

17. Clark–West adjustment intuition

When the larger nested model estimates parameters that are zero under the null, its raw MSPE can be biased upward relative to the parsimonious model.

Clark–West adjusts the loss difference for this extra estimation noise before testing predictive improvement.

Falsifier: check whether one model is algebraically a special case of the other before selecting a DM test automatically.

18. Parameter estimation uncertainty

Forecasts are often produced by models re-estimated through rolling or expanding windows.

The sequence of forecast errors therefore contains uncertainty from both future shocks and parameter estimation.

West (1996) and Giacomini–White develop frameworks that explicitly address forecast evaluation with estimated parameters and conditional predictive ability.

19. Conditional predictive ability

Unconditional DM asks:

Was one model better on average?

But a model can be better only in high-volatility periods.

Giacomini–White tests allow conditional questions such as whether current information predicts which forecast will perform better.

This page keeps the narrower DM owner: average relative predictive loss under a defined evaluation sample.

20. Heteroskedasticity

Financial forecast loss is often more volatile during stress.

Harvey, Leybourne and Newbold showed that ARCH effects can distort forecast-evaluation tests, and newer DM-type research continues to develop heteroskedasticity-robust procedures.

Falsifier: plot d_t variance through time and compare standard DM with heteroskedasticity-aware or bootstrap alternatives when volatility clustering is strong.

21. Inputs and outputs

Inputs can include:

• realised target y_t;
• forecast series 1 and 2;
• forecast horizon h;
• loss function;
• evaluation dates;
• HAC bandwidth/kernel;
• small-sample correction flag;
• one/two-sided alternative;
• nested-model flag;
• missing forecast policy;
• rolling/expanding estimation design;
• bootstrap specification if used.

Outputs can include:

• mean loss for each model;
• mean loss differential d̄;
• DM statistic;
• small-sample corrected statistic;
• HAC standard error;
• p-value;
• confidence interval for E[d];
• loss-differential autocorrelation;
• subperiod statistics;
• robustness across loss/bandwidth choices.

22. Evidence polarity

Evidence for a robust difference includes:

• the sign of d̄ is stable across nearby evaluation windows;
• reasonable HAC choices give similar inference;
• the result survives small-sample correction;
• performance improvement is not concentrated in one or two dates;
• the chosen loss matches the forecasting objective;
• nested-model alternatives confirm the conclusion when required;
• subperiod results are economically interpretable rather than random sign flips;
• effect size is material, not only statistically detectable.

Evidence against confidence includes:

• p-values swing with small bandwidth changes;
• one outlier drives d̄;
• horizon overlap is ignored;
• model nesting is overlooked;
• loss function was selected after seeing results;
• evaluation sample was cherry-picked;
• multiple models were searched but only one DM comparison reported;
• statistically significant loss reduction is economically trivial.

23. Counterexample: same mean loss, different risk of error

Two models can have equal expected squared loss but different error distributions.

One may have stable moderate misses; another rare catastrophic misses.

A DM test under squared loss can correctly find no mean-loss difference while a tail-sensitive decision maker would strongly prefer one model.

Falsifier: inspect error distribution and evaluate a loss function aligned with the actual forecasting job.

24. Counterexample: overlapping horizon ignored

Daily 20-day forecasts create highly overlapping outcome windows.

If the standard error is calculated as if d_t were independent, the test can massively overstate effective sample size.

Falsifier: compare naive and HAC standard errors. A large gap reveals dependence that must be respected.

25. Counterexample: nested predictor model

A baseline forecast is nested inside a larger machine-learning linear model with one extra predictor.

A standard DM statistic reports no improvement because the larger model’s estimation noise inflates its MSPE.

Falsifier: apply Clark–West or appropriate nested-model tests before concluding the extra predictor has no predictive content.

26. Counterexample: one crisis dominates

Model 2 wins overwhelmingly during a five-day crisis and is slightly worse on 500 calm days.

Average squared loss may strongly favour model 2.

That could be exactly right—or misleading if the intended job is ordinary-day forecasting.

Falsifier: report contribution of each date and predefined regime/subperiod results rather than hiding all heterogeneity inside one average.

27. Counterexample: data snooping across many models

If 500 candidate models are tested against a benchmark and only the lowest p-value is reported, ordinary DM significance is invalidated by search.

Alternatives include White’s Reality Check, Hansen’s Superior Predictive Ability test and Model Confidence Set procedures.

Falsifier: preserve the full model-search universe and adjust for multiple comparisons.

28. Counterexample: revised data

A macro-financial forecast may have been made using real-time data that were later revised.

Evaluating the forecast against a revised information set can create a hindsight advantage that did not exist when the forecast was produced.

Falsifier: archive forecast vintages and real-time information sets.

29. Statistical significance is not economic value

A tiny average loss improvement can become highly significant with millions of observations.

That does not tell whether the improvement changes risk limits, pricing error, service levels, capital decisions or another operational outcome.

Always report effect size:

Δloss = d̄

alongside the test statistic.

30. DM versus Brier scoring

Brier-score algorithms define a proper loss for binary probability forecasts and decompose forecast quality.

DM can compare the Brier-loss sequence from two probability models. Brier defines the loss; DM tests the mean difference.

31. DM versus general model validation

Bank model-validation algorithms own the full governance job: conceptual soundness, implementation, benchmarking, outcomes and monitoring.

DM is one statistical challenger-comparison tool inside that wider process.

32. A minimal robust workflow

1. Freeze target, horizon and evaluation sample.
2. Freeze the loss function before viewing results.
3. Generate truly out-of-sample forecasts.
4. Compute d_t.
5. Plot d_t, autocorrelation and outliers.
6. Choose dependence-appropriate long-run variance estimator.
7. Check small-sample correction.
8. Check model nesting.
9. Run the correct alternative test if nested.
10. Report d̄, uncertainty and p-value.
11. Stress loss function, bandwidth and subperiods.
12. Correct for multiple testing if many models were searched.

33. Weak links

• wrong forecast alignment;
• incorrect horizon h;
• look-ahead or revised-data leakage;
• loss chosen after results;
• HAC bandwidth ignored;
• overlapping errors treated independent;
• nested models tested with plain DM;
• outliers silently deleted;
• multiple comparisons ignored;
• sign convention misread;
• p-value reported without effect size.

34. What would falsify confidence?

Confidence should be withdrawn if forecast alignment is wrong; if inference disappears under reasonable HAC choices; if nested-model tests contradict the standard DM result; if the loss function is not appropriate to the decision; if the result is entirely driven by a tiny set of dates; or if multiple-model search makes the reported comparison non-pre-specified.

35. Verification and update triggers

Preserve forecast vintages, target series, horizon, loss function, evaluation dates, d_t, HAC method, bandwidth, small-sample correction, alternative direction, nesting assessment and all model-search metadata.

Revalidate when:

• forecast horizon changes;
• loss objective changes;
• evaluation sample expands materially;
• new regimes appear;
• models become nested/non-nested after redesign;
• data definitions are revised;
• heteroskedasticity or serial dependence increases;
• the model universe expands.

36. Primary and high-quality references

• Francis X. Diebold and Roberto S. Mariano, Comparing Predictive Accuracy, Journal of Business & Economic Statistics, 1995.
• Francis X. Diebold and Roberto S. Mariano, Comparing Predictive Accuracy, NBER Technical Working Paper 169.
• David I. Harvey, Stephen J. Leybourne and Paul Newbold, Testing the Equality of Prediction Mean Squared Errors, International Journal of Forecasting, 1997.
• Kenneth D. West and Todd Clark, Approximately Normal Tests for Equal Predictive Accuracy in Nested Models, NBER Technical Working Paper 326; later Journal of Econometrics.
• Raffaella Giacomini and Halbert White, Tests of Conditional Predictive Ability, Econometrica, 2006.
• David I. Harvey, Stephen J. Leybourne and Paul Newbold, Tests for Equal Forecast Accuracy under Heteroskedasticity, Journal of Applied Econometrics, 2024.

Educational boundary: The Diebold–Mariano framework tests average relative forecast loss under a defined sample and loss function. It does not certify a forecasting model as correct, profitable, safe or appropriate for every regime.