How Realized-Kernel Algorithms Estimate Volatility from Noisy High-Frequency Prices: Autocovariances, Kernel Weights, Bandwidth Choice, Microstructure Noise and Sampling Failure

Reader question: If more intraday price observations should contain more information, why can realised volatility become worse when we sample transactions too frequently?

High-frequency prices contain market-microstructure noise: bid–ask bounce, price discreteness, asynchronous updates, temporary liquidity effects and other deviations between observed transaction/quote prices and the latent efficient price. A naive sum of squared ultra-high-frequency returns can therefore estimate noise as if it were genuine volatility.

Realized-kernel algorithms repair this problem by combining return autocovariances across nearby lags with carefully chosen kernel weights. The zero-lag realised variance is corrected using the serial-dependence pattern that market noise creates.

This article owns one exact computational job: estimating ex-post integrated price variation from noisy high-frequency observations. It does not own GARCH forecasting, OHLC range estimators, market-impact measurement or trade-sign classification. Those are separate mechanisms already represented in the Bukit Timah Tutor finance-and-banking-algorithms estate.

This is public mathematical and computational education. It is not financial advice, a trading signal or a claim that a realised-volatility estimator predicts future returns.

1. Efficient price versus observed price

A common conceptual model writes the observed log price as:

Y_t = X_t + ε_t,

where:

• X_t is the latent efficient log price;
• ε_t is market-microstructure noise.

The efficient price might follow a continuous semimartingale such as:

dX_t = μ_tdt + σ_tdW_t.

The object we often want for a day is its integrated variance:

IV = ∫ σ_t² dt.

If observations were noise-free and sampled increasingly finely, the sum of squared efficient-price returns would converge to quadratic variation. With noisy observed prices, that clean limit breaks.

2. Naive realised variance

Given n intraday observations:

r_i = Y_{t_i} − Y_{t_{i−1}},

the usual realised variance is:

RV = Σ_i=1ⁿ r_i².

At moderate sampling frequencies this can be a good estimator. But because:

r_i = ΔX_i + ε_i − ε_i−1,

the noise difference enters every return.

As the time interval shrinks, true efficient-price returns become smaller, while microstructure noise need not shrink proportionally. The noise-to-signal ratio can therefore increase.

3. Bid–ask bounce creates negative return autocorrelation

Imagine an efficient price that barely moves while observed trades alternate between bid and ask.

A trade at the ask creates a positive observed return; a later trade at the bid creates a negative one. The next return often reverses the previous return.

This creates negative first-order autocovariance in observed high-frequency returns.

Naive RV uses only squared returns and interprets this back-and-forth as genuine variation. A realized kernel also looks at nearby autocovariances and can offset part of that artificial variation.

4. Autocovariances are the building blocks

Define a sample return autocovariance at lag h schematically as:

γ̂_h = Σ r_ir_i−h.

For h = 0:

γ̂₀ = Σ r_i² = RV.

A realized kernel combines γ̂₀ with positive and negative lags:

RK = γ̂₀ + Σ_h=1^H k(h/H)(γ̂_h + γ̂_−h),

where k(·) is a kernel function and H is a bandwidth.

The exact finite-sample definition depends on end effects, jittering and implementation details, but the mechanism is the same: use the dependence pattern across adjacent high-frequency returns to correct the noisy zero-lag variance.

5. Why kernel weights appear

Autocovariances far from lag zero are estimated noisily. Simply summing many lags with equal weight would add excessive sampling variance.

A kernel assigns high weight near zero and progressively lower weight to more distant lags.

The design problem resembles long-run variance estimation in time-series statistics: capture enough serial dependence to correct bias without allowing noisy long-lag estimates to dominate.

6. Bandwidth H is the key bias–variance control

If H is too small, the estimator may not absorb enough noise-induced serial dependence. Residual microstructure contamination remains.

If H is too large, many noisy autocovariances enter the estimator. Sampling variance rises and local price trends or other nonstationary effects can contaminate the estimate.

So:

small H → insufficient noise correction;

large H → excessive variance and contamination.

Realized-kernel theory derives data-dependent bandwidth rules from the relative magnitudes of efficient-price variation and noise. In a production implementation, the estimated bandwidth should be recorded as an output, not hidden as a tuning constant.

7. Noise-to-signal estimation

A practical bandwidth selector needs an estimate of microstructure-noise variance relative to efficient-price variance.

One simple diagnostic compares very high-frequency return variation with lower-frequency realised variance. More sophisticated estimators use return autocovariances and bias corrections.

If estimated noise increases, the optimal realized-kernel bandwidth typically increases because more lag information is needed to remove serial noise effects.

Falsifier: if estimated noise doubles but the selected bandwidth never changes, inspect whether the implementation is actually data adaptive.

8. Kernel choice

Realized-kernel theory allows different weighting functions subject to regularity conditions. Practical work often uses smooth positive-semidefinite kernels because they provide favourable finite-sample and asymptotic behaviour.

A valid implementation should document:

• kernel family;
• kernel support;
• bandwidth;
• endpoint treatment;
• whether the resulting estimator is guaranteed nonnegative under the chosen construction.

The kernel name alone is not enough. Two implementations using the same named kernel can differ materially through bandwidth and boundary handling.

9. End effects and jittering

High-frequency estimators face edge problems at the start and end of the trading window. Individual opening or closing observations can contain larger microstructure effects, auction transitions or timestamp irregularities.

Realized-kernel implementations can use jittering: average a small group of observations near each endpoint to reduce the influence of one noisy first or last price.

This is a numerical correction, but it is also a data-governance decision. The number of endpoint observations and averaging rule should be reproducible.

10. Irregularly spaced transactions

Trades do not arrive every fixed number of seconds. Some periods have many observations; others have none.

The realized-kernel framework can be implemented with transaction-time or endogenously spaced data, but the sampling scheme becomes part of the estimator definition.

Common choices include:

• all trades;
• regular calendar-time sampling;
• previous-tick interpolation;
• refresh-time ideas in multivariate settings;
• quote-midpoint sampling.

Changing the sampling design can change the noise process and therefore the selected bandwidth.

11. Trades versus quotes

Trade prices are exposed to bid–ask bounce and trade-direction effects. Quote midpoints reduce some bid–ask bounce but introduce quote-update and stale-quote issues.

Barndorff-Nielsen, Hansen, Lunde and Shephard studied realized kernels with both trade and quote data and found substantial agreement in practice under careful implementation, while also identifying challenging local trends.

Diagnostic: compute the estimator independently from trade and midpoint data where both are reliable. Large unexplained disagreement is evidence of data or microstructure sensitivity.

12. Inputs and outputs

Inputs can include:

• timestamped transaction or quote prices;
• trading-session boundaries;
• corporate-action adjustments;
• bad-tick filters;
• sampling scheme;
• kernel family;
• noise estimator;
• bandwidth rule;
• endpoint jittering rule;
• overnight-treatment policy.

Outputs can include:

• realized-kernel variance;
• realized-kernel volatility;
• selected bandwidth H;
• noise-variance estimate;
• return autocovariances;
• comparison with ordinary RV at multiple sampling frequencies;
• trade-versus-quote comparison;
• data-quality flags;
• kernel/bandwidth sensitivity;
• rolling stability diagnostics.

13. The volatility signature plot

A useful diagnostic computes ordinary realised variance at many sampling intervals—for example, very fine sampling through increasingly coarse sampling.

With microstructure noise, RV often rises sharply as sampling becomes ultra-fine.

A realized-kernel estimate should be much less sensitive to this extreme-frequency distortion.

Falsifier: if the kernel estimate tracks the exploding ultra-fine RV almost one-for-one, the noise correction is not doing its intended job.

14. Evidence polarity

Evidence for confidence includes:

• estimates are stable across nearby high-frequency sampling schemes;
• trade- and quote-based estimates broadly agree after cleaning;
• kernel volatility does not explode as observation frequency increases;
• selected bandwidth responds sensibly to estimated noise;
• alternative valid kernels produce similar economic conclusions;
• daily estimates reconcile with coarser realised measures when noise is low;
• data-quality exclusions do not drive the result.

Evidence against confidence includes:

• large changes under tiny timestamp/sampling changes;
• negative or erratic estimates under a supposedly nonnegative construction;
• bandwidth values at hard-coded limits;
• trade and quote estimates diverging persistently;
• local intraday trends producing unexplained spikes;
• results dominated by open/close ticks;
• large day-to-day jumps traced to data-feed changes rather than markets.

15. Counterexample: more observations make RV worse

Suppose the efficient price is almost unchanged for a minute, but trades alternate bid–ask–bid–ask.

Sampling every trade records repeated positive and negative returns. Their squares accumulate even though the efficient price barely moved.

Sampling every minute can remove much of this bounce.

Falsifier: compare RV as frequency increases. A steep high-frequency rise indicates that “more data” is adding microstructure noise, not only signal.

16. Counterexample: bandwidth too small

If the true noise has serial dependence over several lags but H = 1, most noise autocovariance remains outside the correction.

Falsifier: increase H across a diagnostic grid. If the estimate falls systematically before stabilising, the original bandwidth was under-correcting noise.

17. Counterexample: bandwidth too large

Suppose intraday prices trend locally for ten minutes because of an information event. Very long-lag autocovariances can start reflecting genuine price dynamics or local nonstationarity rather than microstructure noise.

An overly wide kernel can then remove real variation as if it were noise.

Falsifier: stress H upward. If the estimate continues to change materially rather than forming a stable region, the bias–variance trade-off is unresolved.

18. Counterexample: bad timestamps

A data feed can contain out-of-order trades or timestamps rounded to coarse intervals. Returns calculated in the wrong sequence generate artificial reversals and autocovariances.

Falsifier: verify monotone timestamps, exchange sequence numbers where available, and duplicate-message handling before interpreting return dependence.

19. Counterexample: corporate action

An unadjusted split or special distribution can create an enormous apparent intraday return.

No kernel bandwidth can convert a data-definition error into valid volatility.

Falsifier: reconcile large returns to corporate-action and reference-data events before statistical filtering.

20. Counterexample: overnight information is omitted

Many realised-kernel implementations estimate variation only during a trading session. A stock can close at 100 and open next day at 110 after overnight news.

The intraday kernel may be perfectly estimated while missing the overnight component.

Falsifier: state clearly whether the target is intraday integrated variance or full close-to-close variation. Add overnight returns under a documented convention when the latter is intended.

21. Realized kernel versus Yang–Zhang

Yang–Zhang volatility algorithms use daily open/high/low/close information and explicitly address overnight gaps and drift.

Realized kernels instead use dense intraday returns and target the noise problem that appears when price observations become very frequent.

The two are not replacements for one another; they use different information sets and solve different statistical problems.

22. Realized kernel versus GARCH/EWMA

GARCH and EWMA are primarily conditional forecasting models: past shocks are used to estimate future/conditional variance.

A realized kernel is principally an ex-post measurement estimator for how much efficient-price variation occurred over the observed interval.

Realized measures can later be used as inputs or targets in forecasting models, but measurement and forecasting remain distinct jobs.

23. Connections to microstructure pages

Roll bid–ask-spread algorithms use return covariance to infer transaction-cost information. The realized kernel uses autocovariance differently: as part of a noise-robust variance estimator.

Lee–Ready trade signing highlights why timestamp alignment and quotes matter in high-frequency data pipelines.

24. Alternatives

Sparse-sampled RV: reduce noise by sampling less frequently, but discard observations and require an arbitrary interval choice.

Two-scale/multi-scale realised volatility: combine multiple sampling grids to correct noise.

Pre-averaging: locally averages returns before quadratic-variation estimation.

Subsampling: averages estimates across offset grids.

State-space/parametric noise models: explicitly estimate latent efficient price and observation noise.

The correct method depends on the data-generating assumptions and intended inference.

25. Weak links

• bad-tick and duplicate-message handling;
• timestamp ordering;
• unknown trade/quote conventions;
• opening/closing auction treatment;
• hard-coded bandwidth;
• wrong kernel normalisation;
• overnight omission;
• corporate-action adjustment errors;
• changes in exchange tick size or market structure;
• using one noise model across instruments with radically different liquidity.

26. What would falsify confidence?

Confidence should be withdrawn if estimates remain highly frequency-dependent; if bandwidth changes produce no stable region; if trade- and quote-based estimates disagree materially without explanation; if autocovariance structure changes after data-cleaning corrections; if high-frequency estimates cannot reconcile with known simulation benchmarks; or if results are driven by opening/closing or obviously erroneous observations.

27. Verification and update triggers

Preserve raw market-data identifiers, timestamp precision, cleaning rules, session definition, sampling scheme, kernel, bandwidth, noise estimate, endpoint treatment and all diagnostic comparisons.

Revalidate when:

• data vendor/feed changes;
• exchange microstructure changes;
• tick size changes;
• liquidity regime changes materially;
• trade-versus-quote disagreement rises;
• bandwidth hits imposed limits;
• new asset classes or trading sessions are added;
• overnight/session definitions change.

28. Primary and high-quality references

• Ole E. Barndorff-Nielsen, Peter R. Hansen, Asger Lunde and Neil Shephard, Designing Realized Kernels to Measure the ex post Variation of Equity Prices in the Presence of Noise, Econometrica, 2008.
• Barndorff-Nielsen, Hansen, Lunde and Shephard, Realized Kernels in Practice: Trades and Quotes, The Econometrics Journal, 2009.
• Peter R. Hansen and Asger Lunde, Realized Variance and Market Microstructure Noise, Journal of Business & Economic Statistics.
• Neil Shephard, institutional publication page for Designing Realised Kernels.
• Federal Reserve, Supervisory Guidance on Model Risk Management, for validation and monitoring principles.

Educational boundary: A realized kernel is a statistical estimator of ex-post price variation under explicit assumptions about high-frequency data and noise. It does not turn noisy market data into certainty, and realised volatility is not itself a directional forecast.