Reader question: If we try hundreds or thousands of trading rules, forecasting models or factor definitions and report only the best backtest, how do we test whether the winner is genuinely better than a benchmark rather than simply the luckiest model in a large search?

White’s Reality Check and Hansen’s Superior Predictive Ability (SPA) test treat the entire model search as the object of inference. Instead of testing the selected winner as though it had been chosen in advance, they compare the maximum observed performance across the whole candidate family with a bootstrap null distribution that preserves the dependence structure among strategies and through time.

This article owns one precise computational job: benchmark-based multiple-comparison testing for a family of financial forecasts or strategies after data snooping. It does not own false-discovery-rate control across independent hypotheses, single-pair Diebold–Mariano testing, Sharpe-ratio estimation, strategy design or general cross-validation.

This is public mathematical and computational education. It is not investment advice and does not turn a statistically significant backtest into a deployable trading strategy.

1. The winner’s curse begins before the test

Suppose we generate 1,000 strategies whose true expected excess return over a benchmark is exactly zero.

Even if every strategy is noise, one of them will usually have the best historical Sharpe ratio, lowest forecast loss or highest average return.

If we now run an ordinary one-strategy t-test only on that winner, we ignore the fact that it was selected because its sample result was unusually favourable.

The relevant null is not:

“Is this one strategy’s mean differential zero?”

It is:

“Could the best result among all strategies in the search have arisen when none truly beats the benchmark?”

2. Define a common performance differential

Let model 0 be the benchmark and k = 1,…,K be competing models.

Let L_0,t and L_k,t be losses at time t, where smaller is better.

Define:

d_k,t = L_0,t − L_k,t.

Then:

d_k,t > 0 means competitor k beat the benchmark at time t;
d_k,t < 0 means the benchmark won.

The sample mean differential is:

d̄_k = (1/T) Σ_t=1^T d_k,t.

3. The family null

The benchmark-based null can be written:

H₀: max_k E[d_k,t] ≤ 0.

That is: no competitor has positive expected advantage over the benchmark.

The alternative is:

H₁: at least one competitor has E[d_k,t] > 0.

This is a family-level statement. Rejecting H₀ establishes evidence that something in the candidate family beats the benchmark; it does not automatically identify every superior strategy.

4. White’s Reality Check statistic

A simple Reality Check statistic is based on the best sample mean:

T_RC = max_k √T d̄_k.

The maximum is the key. If K increases, the null distribution of the maximum also moves upward because more opportunities exist for lucky sample performance.

This is exactly the multiplicity effect that a one-model test misses.

5. Why ordinary iid resampling is usually wrong

Strategy and forecast differentials can exhibit:

serial correlation;
volatility clustering;
overlapping holding periods;
position persistence;
common shocks across strategies;
strong cross-strategy correlation.

Resampling individual observations independently destroys this structure and can produce a null distribution that is too narrow.

White’s framework commonly uses a dependent-data bootstrap such as the stationary bootstrap, which resamples random-length time blocks.

6. Stationary bootstrap mechanics

In the stationary bootstrap, each resampled observation either:

continues to the next original time point with probability 1−p; or
starts a new block at a random original index with probability p.

The block length is geometric with:

E[block length] = 1/p.

Long blocks preserve more serial dependence but reduce the effective number of independent pieces. Short blocks give more resampling diversity but can destroy dependence.

Block length is therefore a genuine tuning parameter, not a cosmetic setting.

7. Recentering creates the bootstrap null

Suppose the observed best strategy has a positive sample mean. If bootstrap samples are drawn directly from the uncentered differential series, that positive sample advantage remains embedded in the resamples.

To simulate a null in which expected superiority is absent, Reality Check bootstrap statistics are constructed from recentred performance differentials.

Schematically, for bootstrap replicate b:

T_b^* = max_k √T (d̄_k,b^* − d̄_k).

The p-value is the fraction of bootstrap maxima at least as large as the observed maximum.

Without correct centering, the bootstrap can answer the wrong hypothesis while still returning plausible-looking numbers.

8. Preserve cross-strategy dependence

When a time block is resampled, the same time indices should be applied across all K strategies.

If each strategy is bootstrapped independently, the resampling destroys cross-strategy correlation.

That is especially damaging when strategies are parameter variations of the same idea and therefore highly correlated.

Falsifier: verify that every bootstrap replicate contains one common sequence of resampled time indices applied to the entire T×K differential matrix.

9. Why many poor alternatives can hurt White’s test

Imagine one genuinely promising strategy and 999 catastrophically bad ones.

White’s unstudentized maximum test can become conservative because the candidate family contains noisy, irrelevant alternatives whose sampling behaviour influences the bootstrap maximum.

Peter Hansen developed the SPA test to improve power and reduce sensitivity to poor alternatives.

10. Hansen’s first modification: studentization

Strategies can have very different variance.

A mean differential of 0.10 with standard deviation 10 is less persuasive than a mean differential of 0.08 with standard deviation 0.20.

The SPA statistic therefore uses a standardized performance differential:

t_k = √T d̄_k/ω̂_k,

where ω̂_k estimates the long-run standard deviation of the differential series.

The family statistic is based on the best standardized candidate, typically with a zero floor so clearly negative competitors do not define evidence of superiority.

11. Hansen’s second modification: sample-dependent recentering

The SPA bootstrap does not force every clearly inferior model to sit exactly on the boundary of the null.

Instead, sample evidence is used to reduce the influence of models that are far below the benchmark while retaining appropriate treatment of models that could plausibly lie near the null boundary.

This produces better power than treating a huge collection of obviously poor alternatives as equally relevant competitors.

12. Long-run variance estimation matters

Studentization requires an estimate of:

Var(√T d̄_k).

With serial dependence, the ordinary sample variance of d_k,t is insufficient. Long-run variance incorporates autocovariances.

Possible implementations use:

bootstrap variance;
HAC-style estimators;
stationary-bootstrap-based estimates.

A model with nearly zero estimated variance can create numerical explosions in t-statistics, so minimum-variance safeguards and diagnostics are essential.

13. Inputs and outputs

Inputs can include:

benchmark loss or return series;
K competing model loss/return series;
common evaluation dates;
loss definition;
transaction-cost treatment;
bootstrap method;
mean block length;
number of bootstrap replications B;
SPA recentering choice;
long-run variance estimator;
random seed.

Outputs can include:

best observed candidate;
mean performance differentials;
Reality Check statistic and p-value;
SPA statistic and p-value(s);
candidate t-statistics;
bootstrap null quantiles;
block-length sensitivity;
Monte Carlo p-value error;
family size K;
candidate correlation diagnostics.

14. A minimal Reality Check algorithm

Define the complete candidate family before testing.
Compute the T×K differential matrix d.
Compute d̄_k for every candidate.
Set T_obs = max_k √T d̄_k.
Generate one dependent-data bootstrap index sequence.
Apply it to all K columns jointly.
Compute recentred bootstrap means.
Record the bootstrap maximum.
Repeat B times.
Estimate p-value as the bootstrap exceedance fraction.

15. A minimal SPA algorithm

Start with the same differential matrix.
Estimate long-run standard deviation ω̂_k for every candidate.
Compute standardized observed performance.
Apply Hansen’s sample-dependent null recentering so very poor models receive less influence.
Bootstrap the entire candidate family jointly through time.
Compute standardized bootstrap maxima.
Estimate SPA p-values and sensitivity to recentering/block choices.

16. The candidate universe must be honest

A multiple-testing correction only protects against the model family that is actually included.

If a researcher tried 10,000 variants, discarded 9,900 silently and runs Reality Check on the surviving 100, the multiplicity adjustment is incomplete.

Falsifier: preserve the full strategy-generation log or, at minimum, a defensible reconstruction of the search universe.

17. Transaction costs belong inside the differential

If one strategy trades 50 times more often than the benchmark, gross returns can create a false advantage.

Performance differential should reflect the intended economic loss function, including documented:

commissions;
bid–ask spread;
market-impact assumptions;
financing/borrow costs;
turnover penalties where relevant.

Statistical correction for data snooping cannot repair an economically incomplete backtest.

18. The test depends on the loss function

A volatility forecast can be judged by squared error, QLIKE or another loss.

A trading strategy can be judged by return differential, utility or risk-adjusted loss.

Different loss functions can produce different winners and different nulls.

Falsifier: pre-specify the performance criterion and run sensitivity to credible alternatives rather than choosing the loss after seeing which one rejects.

19. Reality Check and SPA are benchmark tests, not best-model selectors

Rejecting H₀ tells us that the benchmark is inferior to at least one candidate in the tested family.

It does not prove that the sample winner is uniquely best in population.

If the question is “which models cannot be statistically distinguished from the best?”, a Model Confidence Set is more directly aligned.

20. Evidence polarity

Evidence for a genuine family-level advantage includes:

small RC/SPA p-values across reasonable block lengths;
similar conclusions under several credible loss definitions;
performance survives realistic transaction costs;
the best candidate remains strong out of sample;
results are not driven by one short crisis period;
candidate universe is fully documented;
bootstrap replication error is small;
SPA remains significant after poor alternatives are downweighted.

Evidence against confidence includes:

naive winner t-test significant but RC/SPA not significant;
p-value changes sharply with block length;
significance disappears after costs;
candidate family was reconstructed after selection;
one or two observations create the result;
different credible loss functions reverse the conclusion;
stationarity breaks across the evaluation sample;
out-of-sample performance collapses.

21. Counterexample: 1,000 identical parameter tweaks

If all 1,000 strategies are almost perfectly correlated, treating them as 1,000 independent trials would be far too harsh.

Joint block bootstrap preserves their dependence and therefore allows the effective multiplicity penalty to reflect the fact that many candidates are nearly duplicates.

Falsifier: compare the empirical candidate correlation structure with a simulation that incorrectly bootstraps columns independently.

22. Counterexample: 999 terrible strategies reduce power

White’s RC can become conservative when many poor or high-variance alternatives are added.

Falsifier: compare RC and Hansen SPA on the same family. A large power difference is a clue that irrelevant alternatives materially affect the RC null.

23. Counterexample: wrong bootstrap blocks

A one-day block length is used for a strategy with five-day overlapping returns.

The bootstrap destroys the overlap dependence and understates null variation.

Falsifier: increase block length and verify whether p-values stabilise.

24. Counterexample: only the winner is bootstrapped

The researcher first selects the best strategy, then bootstraps that one series.

This removes the selection process from the null distribution—the exact source of data-snooping bias.

Falsifier: each bootstrap replicate must recompute the maximum across all candidates.

25. Counterexample: strategy family changed after seeing results

A researcher sees that momentum works, then adds 500 momentum variants but omits 500 failed value variants tried earlier.

The test now conditions on a selectively curated family.

Falsifier: preserve experiment provenance: candidate-generation rules, timestamps and rejected variants.

26. Counterexample: structural break

Suppose a 20-year sample combines two different market structures. The bootstrap assumes the observed dependence pattern is a meaningful basis for resampling.

If strategy profitability changes permanently after a regulation or market-design shift, a stationary bootstrap over the full period can mix incompatible regimes.

Falsifier: run regime/subperiod diagnostics and out-of-sample validation. Resampling cannot make structural instability disappear.

27. Counterexample: huge B is not enough if the data are weak

Increasing bootstrap replications from 1,000 to 100,000 reduces simulation noise in the p-value.

It does not create more historical information.

Falsifier: distinguish bootstrap Monte Carlo error from sampling uncertainty in the original T observations.

28. Relationship to Diebold–Mariano

Diebold–Mariano algorithms own the single-pair equal-predictive-accuracy problem.

Reality Check and SPA own a different question:

does any member of a searched family beat a benchmark after accounting for the family search?

29. Relationship to Benjamini–Hochberg

Benjamini–Hochberg algorithms control expected false discovery proportion across many p-values.

White/SPA instead work with a maximum benchmark-comparison statistic and a joint bootstrap distribution. Their inferential targets differ.

FDR control may be useful when identifying a set of discoveries; SPA is useful when asking whether a benchmark is beaten at all.

30. Relationship to Deflated Sharpe Ratio

The Deflated Sharpe Ratio adjusts an observed Sharpe for multiple trials and non-normal return features under a different modelling framework.

Reality Check/SPA are more general in the loss function and can preserve serial/cross-strategy dependence through bootstrap, but they are computationally heavier and depend on resampling validity.

31. Weak links

incomplete candidate universe;
incorrect recentring;
independent column bootstrap;
block length too short or too long;
too few bootstrap replications;
loss function selected after looking at results;
transaction costs omitted;
near-zero variance candidates causing unstable studentization;
structural breaks ignored;
winner interpreted as uniquely best after family-level rejection.

32. What would falsify confidence?

Confidence should be withdrawn if the family-level p-value is not robust to reasonable resampling choices; if the candidate universe is incomplete; if costs eliminate the advantage; if significance is concentrated in one unstable subperiod; if the winner fails out of sample; or if a reproducible implementation of the joint bootstrap does not match the reported result.

33. Verification and update triggers

Preserve the complete candidate list, benchmark, loss function, differential matrix, bootstrap method, block parameter, number of replications, random seed, variance estimator and all recentering rules.

Revalidate when:

new strategies are added to the search universe;
transaction-cost assumptions change;
forecast horizon changes;
market structure changes;
loss function changes;
bootstrap dependence diagnostics change;
the benchmark changes;
strategy-generation code changes.

34. Primary and high-quality references

Halbert White, A Reality Check for Data Snooping, Econometrica, 2000.
Peter R. Hansen, A Test for Superior Predictive Ability, Journal of Business & Economic Statistics, 2005.
Ryan Sullivan, Allan Timmermann and Halbert White, Data-Snooping, Technical Trading Rule Performance, and the Bootstrap, Journal of Finance.
Peter R. Hansen and Asger Lunde, A Forecast Comparison of Volatility Models: Does Anything Beat a GARCH(1,1)?, an empirical RC/SPA application.
Martin Politis and Joseph Romano, stationary-bootstrap research underlying dependent-data resampling.

Educational boundary: RC/SPA significance is evidence against a family-level null relative to a stated benchmark and stated search universe. It is not proof that the selected strategy will remain profitable, that its economic costs are modelled correctly, or that the search process itself was fully disclosed.

Small Group Tutorials

How White Reality-Check and Hansen SPA Algorithms Test Financial Strategies After Data Snooping: Bootstrap Maxima, Recentering, Studentization, Poor Alternatives and False Discovery Failure