Reader question: If a researcher tests hundreds or thousands of financial signals, factors, forecasting rules or model variants, how can statistically significant results be separated from discoveries that appear simply because so many hypotheses were tried?

The Benjamini–Hochberg (BH) procedure controls the false discovery rate: the expected proportion of false rejections among the hypotheses declared significant. Instead of demanding that the probability of even one false rejection be tiny, BH ranks all valid p-values and applies an increasing threshold that trades stricter multiplicity control against useful statistical power.

This article owns one precise computational job: batch multiple-testing control for a pre-defined family of financial hypotheses using false-discovery-rate procedures. It does not own the construction of trading signals, the validity of each underlying test statistic, pairwise forecast comparison, backtest execution, or causal interpretation.

For example, Diebold–Mariano algorithms own a particular forecast-loss comparison and its HAC/overlapping-horizon issues. BH begins only after a family of valid p-values has been defined.

This is public mathematical and computational education. It is not financial advice, a signal-selection recommendation, or a claim that statistical significance implies economic value or future profitability.

1. The multiplicity problem

Suppose m independent null hypotheses are all true and every test uses a 5% significance level.

The expected number of false rejections is:

0.05m.

If m = 1,000, that is 50 false positives on average.

The probability of at least one false positive under independence is:

1 − (1−0.05)¹⁰⁰⁰,

which is essentially one.

A workflow that reports only the most attractive p < 0.05 result after testing 1,000 ideas has not performed a 5% experiment.

2. Define R and V

Let:

R = total number of rejected null hypotheses;
V = number of those rejections that are actually false discoveries.

The realised false discovery proportion is:

FDP = V / max(R,1).

The false discovery rate is:

FDR = E[FDP].

V is unobservable in real research because we do not know which nulls are truly correct. FDR procedures control this quantity probabilistically under assumptions about the p-values and their dependence.

3. FDR is not family-wise error rate

The family-wise error rate (FWER) is:

P(V ≥ 1).

FWER asks: what is the probability of making even one false rejection?

FDR asks: among the discoveries we make, what fraction should we expect to be false?

When all null hypotheses are true, FDR and FWER are closely related because every rejection is false. When many alternatives are genuinely non-null, FDR can allow substantially more power than procedures designed to make any false rejection very rare.

4. The Benjamini–Hochberg step-up algorithm

Suppose m valid p-values are:

p₁,…,p_m.

Choose a target false discovery rate q, for example q = 0.05.

Sort the p-values: p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p_(m).
For each rank i, compute the BH critical value:

c_i = (i/m)q.

Find the largest k such that:

p_(k) ≤ (k/m)q.

Reject hypotheses corresponding to p₍₁₎,…,p_(k).

If no rank satisfies the inequality, reject none.

5. Why it is a step-up procedure

The threshold becomes less strict as rank i increases:

q/m, 2q/m, 3q/m, …, q.

The algorithm searches upward for the largest acceptable rank, then rejects every smaller p-value as well.

This differs from testing every hypothesis against one Bonferroni threshold α/m.

6. Worked example

Suppose m = 10 and q = 0.05.

The sorted p-values are:

0.001, 0.004, 0.009, 0.018, 0.031, 0.08, 0.20, 0.40, 0.60, 0.90.

The BH thresholds are:

0.005, 0.010, 0.015, 0.020, 0.025, 0.030, 0.035, 0.040, 0.045, 0.050.

Ranks 1–4 satisfy their thresholds; rank 5 does not.

The largest passing rank is k = 4, so the first four hypotheses are rejected.

Note the step-up logic: the algorithm does not stop permanently at the first local failure while scanning upward unless that is how the implementation happens to search for the maximum.

7. BH-adjusted p-values

Instead of rerunning the rank threshold for every q, software often reports BH-adjusted p-values.

For sorted p-values, a raw adjustment is:

a_(i) = (m/i)p_(i).

To enforce monotonicity:

p_adj,(i) = min_j≥i a_(j),

capped at 1.

Then hypothesis i is rejected at FDR target q when:

p_adj,i ≤ q.

An implementation that omits the cumulative minimum can produce adjusted p-values that decrease with worse-ranked hypotheses, which is incorrect.

8. What BH actually guarantees

In the original 1995 result, BH controls FDR at or below q under independence of the relevant test statistics/p-values, with the exact bound involving the proportion of true null hypotheses.

Later work extends BH control to broad forms of positive dependence, often described through positive regression dependency on subsets (PRDS).

This matters because financial tests are rarely independent.

9. Correlated financial hypotheses

Financial signals often share:

the same return sample;
similar predictors;
overlapping holding periods;
common sectors/assets;
nested model specifications;
common macro factors.

The resulting p-values can be strongly dependent.

Positive dependence does not automatically invalidate BH, but arbitrary dependence requires more caution.

10. Benjamini–Yekutieli under arbitrary dependence

Benjamini and Yekutieli provide a conservative modification for arbitrary dependence.

Define:

c(m) = Σ_j=1^m 1/j.

Replace the BH critical values with:

c_i^BY = i q / [m c(m)].

Because c(m) grows roughly like log m + γ_Euler, BY can be substantially more conservative when m is large.

The price of arbitrary-dependence protection is lower power.

11. Dependence is not the only problem: p-values must be valid

BH cannot repair invalid individual tests.

If a p-value is too small because:

serial correlation was ignored;
heteroskedasticity was ignored;
overlapping returns were treated as independent;
parameters were estimated on the same sample without accounting for it;
the test statistic has the wrong null distribution;

then BH is controlling multiplicity over miscalibrated inputs.

Public rule: valid p-values first, multiplicity control second.

12. Overlapping horizons

A 12-month return calculated every month creates heavily overlapping observations.

A naive t-test that treats all monthly overlapping returns as independent will underestimate standard errors.

Applying BH afterward does not fix that underestimation.

This connects to Diebold–Mariano/HAC reasoning: the per-hypothesis statistic needs a dependence-aware variance estimate when its data structure requires one.

13. The hypothesis family must be defined before seeing winners

Suppose a researcher tests 10,000 signals but reports the 50 with the smallest p-values and then applies BH only to those 50.

The multiplicity family has been truncated after selection.

This generally understates the search that produced the winners.

Falsifier: maintain a research ledger of every hypothesis/parameter variant that entered the search family. Multiplicity control should reflect the actual discovery process, not only the survivors.

14. Parameter sweeps create hidden hypotheses

A “single” moving-average strategy may have been tested at:

50 lookback lengths;
20 thresholds;
10 asset universes;
5 volatility filters.

That is up to 50,000 candidate specifications, not one hypothesis.

If the best parameter combination is selected after seeing returns, the search multiplicity must be addressed by an appropriate data-snooping/backtest framework.

BH can be part of that framework only if the hypothesis family and p-values genuinely represent the search.

15. The factor zoo problem

Financial economics has accumulated large numbers of proposed return predictors/factors.

Harvey, Liu and Zhu argue that conventional single-test significance hurdles are inadequate when many factors have been explored and published.

Later work by Harvey examines false and missed discoveries in financial economics and explicitly treats the trade-off between Type I and Type II errors.

The broad lesson matches BH’s motivation: the evidence threshold must account for how many opportunities there were to discover something by chance.

16. FDR target q is not “probability this discovery is false”

If q = 0.05 and a hypothesis is rejected by BH, it is incorrect to say:

“There is only a 5% probability this particular discovery is false.”

FDR is an expectation over the proportion of false discoveries in the entire rejection set under repeated sampling/model assumptions.

It is not an individual posterior probability.

17. FDR does not guarantee FDP ≤ q in every experiment

FDR controls:

E[FDP].

In one realised dataset the false discovery proportion can exceed q.

Other procedures target stronger tail guarantees on FDP, but BH’s standard guarantee is expectation-based.

This distinction matters when the cost of even one false discovery is extremely high.

18. When FWER may be the better target

If a bank makes one irreversible high-stakes governance decision from a family of tests, controlling the probability of any false rejection may be more appropriate than controlling average false-discovery proportion.

Bonferroni or Holm procedures target FWER.

BH is most attractive when:

many discoveries can be useful;
some false discoveries are tolerable at a controlled rate;
power matters materially.

Error-rate choice is a decision-design question, not a universal ranking of procedures.

19. Bonferroni comparison

Bonferroni rejects p_i when:

p_i ≤ α/m.

It controls FWER under very broad dependence conditions.

BH uses larger thresholds for higher-ranked p-values, which can produce many more discoveries when there are genuine non-null effects.

The trade-off is that BH controls a different error quantity.

20. Holm procedure

Holm’s step-down procedure also controls FWER and is uniformly at least as powerful as plain Bonferroni.

It orders p-values and compares them with progressively less stringent thresholds.

The ordering idea can look superficially similar to BH, but the target error rate and threshold sequence differ.

21. Storey q-values and estimating the null proportion

BH does not require estimating the proportion π₀ of hypotheses whose nulls are true.

Adaptive FDR methods, including Storey’s q-value framework, estimate π₀ and can gain power when many hypotheses are genuinely non-null.

That extra power introduces another estimation layer.

Falsifier: if adaptive results depend strongly on the π₀ tuning rule, compare with ordinary BH as a simpler benchmark.

22. Weighted BH

If hypotheses have genuinely pre-specified different prior importance or power, weighted FDR procedures can assign weights before testing.

Weights must not be chosen after observing the same p-values in a way that favours winners.

Post-hoc weights recreate selection bias.

23. Online FDR is a different problem

BH assumes a batch/family of hypotheses is available together.

A live research system may test hypotheses sequentially over time, with the number of future tests unknown.

Online FDR procedures such as alpha-investing/LORD-type methods allocate an error budget sequentially.

This article does not own online FDR; it owns the batch BH family.

24. Data snooping beyond p-value multiplicity

Financial backtests contain other selection effects:

choosing the sample start/end after inspection;
choosing transaction-cost assumptions;
choosing assets that survived;
choosing model form after seeing failures;
discarding unattractive experiments.

BH on the final p-value table cannot reconstruct experiments that were never recorded.

Research process integrity is upstream of any multiple-testing formula.

25. White’s Reality Check and SPA

In strategy/data-snooping settings, White’s Reality Check and Hansen’s Superior Predictive Ability test address the problem of selecting the best model from many alternatives relative to a benchmark under dependence.

These procedures target a different joint hypothesis structure from BH.

Alternative: use a method aligned with the actual research question—“which individual hypotheses survive FDR?” versus “does the best rule outperform after searching many rules?”

26. Deflated Sharpe Ratio

Financial strategy research also uses the Deflated Sharpe Ratio to adjust a Sharpe-ratio discovery for non-normality and multiple trials/selection effects.

It is not an FDR procedure.

It answers a more strategy-specific question about whether an observed Sharpe remains exceptional after accounting for selection and distributional effects.

27. Inputs and outputs

Inputs can include:

pre-defined hypothesis family;
valid p-value per hypothesis;
target q;
BH or BY dependence policy;
weights if a pre-specified weighted method is used;
hypothesis metadata/IDs;
test-statistic and sample-definition lineage.

Outputs can include:

ranked p-values;
BH critical values;
largest passing rank k;
rejection set;
BH-adjusted p-values;
BY-adjusted results where required;
family size m;
dependence diagnostics;
hypothesis-ledger completeness checks;
out-of-sample replication status.

28. Evidence polarity

Evidence for confidence includes:

hypothesis family defined before winner selection;
p-values calibrated under the correct null/data dependence;
BH/BY implementation agrees with trusted statistical software;
adjusted p-values are monotone in sorted order;
discoveries persist under reasonable dependence-aware corrections;
results replicate on held-out data;
economic effect sizes remain material after statistical filtering;
research ledger captures failed/abandoned trials.

Evidence against confidence includes:

family defined after looking at p-values;
individual tests ignore serial correlation/overlap;
discoveries vanish under BY or resampling-based dependence control;
many nearly duplicate specifications dominate the family;
out-of-sample effects collapse;
economic significance is negligible despite tiny p-values;
unreported experiments make m unknowable.

29. Counterexample: BH on invalid t-tests

One thousand overlapping 12-month-return signals are tested with iid standard errors.

The p-values are too small because dependence is ignored.

BH correctly processes the numbers it receives but the discoveries remain invalid.

Falsifier: rebuild each test with HAC/block-bootstrap/inference appropriate to the return construction before applying multiplicity control.

30. Counterexample: select top 100, then BH

Ten thousand factors are tested. The top 100 by t-statistic are saved; the other 9,900 are deleted. BH is applied to the 100.

The family has been conditioned on selection.

Falsifier: recover the full search ledger. If it cannot be recovered, do not claim standard BH FDR control for the post-selected subset.

31. Counterexample: correlated clones

A researcher tests the same momentum idea at lookbacks 120,121,…,250 days.

The hypotheses are highly correlated clones.

Standard BH may still have useful control under certain positive dependence structures, but the family contains less independent scientific information than m suggests.

Diagnostic: cluster near-duplicate hypotheses and report dependence structure. Consider resampling or hierarchical testing when families are strongly structured.

32. Counterexample: all nulls are true

If every signal has zero true effect, every rejection is a false discovery.

BH control then becomes closely connected to controlling the probability/expected occurrence of any rejection.

Falsifier: simulate the exact null dependence structure and verify empirical FDR over many replications.

33. Counterexample: many real effects

If 40% of tested hypotheses are genuinely non-null, Bonferroni may reject only a few very strong effects.

BH can reject more while targeting the expected false fraction among discoveries.

This is the power advantage for which FDR control was designed.

34. Counterexample: one discovery only

If BH returns R = 1, then FDP is either 0 or 1 in that realised study.

An FDR target of 5% does not make the realised false-discovery fraction 5%.

This illustrates why expectation-based guarantees should not be translated into per-discovery probabilities.

35. Counterexample: p-hacking family size is unknown

A strategy idea was tuned manually for months, with undocumented variants.

The final backtest has p = 0.001.

No honest m is available.

Falsifier: BH cannot rescue undocumented search. Use fresh preregistered/held-out data and begin a new auditable test family.

36. Simulation verification

A strong implementation should simulate known null/alternative families.

For each replication:

generate test statistics under specified dependence;
compute valid p-values;
apply BH/BY;
record V, R and FDP;
average FDP across replications.

The empirical FDR should align with the theoretical target under the method’s assumptions.

37. Economic significance remains separate

A factor can have a tiny p-value and an economically negligible expected return.

A forecast can be statistically different but not enough to cover transaction costs.

A credit-model variable can be significant but add no useful out-of-sample discrimination/calibration.

BH controls statistical false discoveries, not business usefulness.

38. Replication is stronger than adjusted significance

A discovery that survives BH in one dataset is more credible if it also survives:

a pre-specified out-of-sample period;
a different market/universe where theory predicts transfer;
reasonable cost assumptions;
alternative valid inference methods.

Multiplicity correction and replication solve different failure modes and should be used together.

40. What would falsify confidence?

Withdraw confidence if null simulations exceed the intended FDR; if p-values are miscalibrated; if the family ledger is incomplete; if discoveries vanish under dependence-aware or resampling procedures; if held-out replication fails; or if the apparent result depends on arbitrary family boundaries chosen after seeing the winners.

41. Verification and update triggers

Preserve every hypothesis ID, test specification, dataset/version, test statistic, raw p-value, family assignment, q target, dependence method, adjusted p-value and rejection decision.

Revalidate when:

new hypotheses are added to the family;
test-statistic inference changes;
dependence structure changes;
data revisions occur;
research search procedures change;
the use changes from exploratory discovery to high-stakes governance;
out-of-sample replication evidence arrives.

42. Primary and high-quality references

Yoav Benjamini and Yosef Hochberg, Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, Journal of the Royal Statistical Society Series B, 1995.
Yoav Benjamini and Daniel Yekutieli, The Control of the False Discovery Rate in Multiple Testing under Dependency, Annals of Statistics, 2001; bibliographic record: EPA HERO.
Yoav Benjamini, Discovering the False Discovery Rate, JRSS B, 2010.
Campbell R. Harvey, False (and Missed) Discoveries in Financial Economics, Journal of Finance, 2020.
Campbell Harvey, Yan Liu and Heqing Zhu, …and the Cross-Section of Expected Returns, Review of Financial Studies, on multiple testing in the factor literature.
Halbert White, A Reality Check for Data Snooping, Econometrica, for a different multiple-model backtest question; Peter Hansen, A Test for Superior Predictive Ability, for the SPA framework.

Educational boundary: Benjamini–Hochberg controls a statistical error rate over a declared family of valid tests. It cannot make an undocumented research search honest, cannot repair invalid p-values, and cannot convert statistical significance into a profitable or economically meaningful financial decision.

Small Group Tutorials

How Benjamini–Hochberg Algorithms Control False Discoveries in Financial Research: Ranked P-Values, FDR Thresholds, Dependent Tests, Factor Zoos and Data-Snooping Failure