How Kalman-Filter Yield-Curve Algorithms Track Dynamic Level, Slope and Curvature: State-Space Models, Prediction–Update Cycles, Measurement Error and Regime Failure

Reader question: A yield curve is observed at many maturities every day, but individual quoted yields contain noise, missing points and market-specific distortions. How can an algorithm infer a small set of hidden curve factors that evolve through time without refitting every day as though yesterday never happened?

A Kalman-filter yield-curve algorithm writes the problem as a state-space model. Hidden factors such as level, slope and curvature evolve through a transition equation. Observed yields are noisy linear measurements of those factors through maturity-specific loadings. The filter predicts today’s state from yesterday, compares the predicted curve with today’s observations, and then updates the hidden factors in proportion to how informative each observation is.

The result is a dynamic curve estimate, a covariance matrix describing uncertainty about the factors, and a natural one-step-ahead forecasting system.

What this page owns — and what it does not

This page owns the dynamic inference transformation:

time series of noisy multi-maturity yields + state-space model → filtered latent factors + uncertainty + forecasts.

It does not replace static Nelson–Siegel–Svensson curve fitting, which owns cross-sectional nonlinear calibration at one date; yield-curve PCA, which owns covariance eigenvectors and empirical factor shocks; or curve bootstrapping, which constructs discount factors and forwards from market instruments.

This is statistical term-structure education, not an interest-rate forecast for trading and not personalized financial advice.

Why one-day-at-a-time curve fitting throws information away

Suppose a three-factor curve is fitted independently every business day.

Monday’s factors may be:

(level, slope, curvature) = (4.0%, −1.2%, 0.5%).

Tuesday’s optimizer then starts from scratch except perhaps for a numerical initial guess.

But economically, Tuesday’s curve is strongly related to Monday’s. A state-space model encodes that persistence directly instead of treating every date as a new unrelated cross-section.

The Dynamic Nelson–Siegel measurement equation

A common three-factor Nelson–Siegel yield representation is:

y_t(τ) = β_1,t + β_2,t[(1−e^−λτ)/(λτ)] + β_3,t{[(1−e^−λτ)/(λτ)]−e^−λτ} + ε_t(τ).

The factors are commonly interpreted as:

• β₁: level;
• β₂: slope;
• β₃: curvature.

The loading shapes depend on maturity τ and decay parameter λ. Diebold and Li showed how these time-varying factors provide a useful dynamic representation of government-bond yield curves.

Why the loadings look like level, slope and curvature

The level loading is one at every maturity, so changing β₁ shifts the whole curve.

The slope loading starts near one at very short maturities and decays toward zero at long maturities, so β₂ mainly changes the short end relative to the long end.

The curvature loading starts near zero, rises at intermediate maturities and falls again, creating a hump-shaped middle-curve effect.

These shapes connect naturally to the empirical PCA observation that much yield-curve variation is often summarized by level-, slope- and curvature-like movements.

State vector notation

Collect the latent factors into:

x_t = [β_1,t, β_2,t, β_3,t]^T.

Collect observed yields at maturities τ₁,…,τ_m into:

y_t = [y_t(τ₁),…,y_t(τ_m)]^T.

Then the yield-curve measurement equation becomes:

y_t = Hx_t + ε_t,

where matrix H contains the Nelson–Siegel maturity loadings.

The state-transition equation

The hidden factors evolve through a dynamic model such as:

x_t = c + Ax_t−1 + η_t,

with:

η_t ~ N(0,Q).

The measurement noise is:

ε_t ~ N(0,R).

The two covariance matrices play different roles:

• Q: how much the true latent factors can move unpredictably between dates;
• R: how noisy observed yields are around the curve implied by the latent factors.

The filter starts with a distribution, not one certain state

At time t−1, the algorithm has:

x̂_t−1|t−1 — the filtered factor estimate;

P_t−1|t−1 — its estimation-error covariance.

The covariance is essential. A Kalman filter does not merely say “the factor is 3.8.” It carries information about how uncertain that estimate is and how factor uncertainties co-move.

Prediction step

Before seeing today’s yields:

x̂_t|t−1 = c + A x̂_t−1|t−1.

The predicted covariance is:

P_t|t−1 = A P_t−1|t−1A^T + Q.

The first equation projects yesterday’s best factor estimate forward. The second admits that uncertainty grows because new process noise can move the true factors.

Predict today’s yield curve

The predicted observed yields are:

ŷ_t|t−1 = Hx̂_t|t−1.

The innovation or measurement residual is:

v_t = y_t − ŷ_t|t−1.

This is the new information arriving from the market: the difference between what the state model expected and what the yield observations actually show.

Innovation covariance

The uncertainty of the innovation is:

S_t = H P_t|t−1H^T + R.

If state uncertainty is large, the curve prediction is uncertain. If measurement noise is large, an observed yield surprise is less informative.

The Kalman filter combines both sources when deciding how strongly to update.

Kalman gain: how much should the market observation move the hidden factors?

The Kalman gain is:

K_t = P_t|t−1H^TS_t⁻¹.

Then:

x̂_t|t = x̂_t|t−1 + K_tv_t.

If measurement noise is tiny relative to prior uncertainty, the gain can place substantial weight on the new yields. If the quote is considered noisy, the filter moves the latent factors less.

This is not an arbitrary smoothing parameter. It is derived from the model’s covariance structure.

Update the state uncertainty

A compact covariance update is:

P_t|t = (I−K_tH)P_t|t−1.

In numerical implementations, a Joseph-form or square-root update can be preferable because it better preserves symmetry and positive semidefiniteness under floating-point rounding.

A covariance matrix that develops negative eigenvalues is a numerical warning, not a new economic discovery.

A one-maturity intuition

Suppose the model predicts a 10-year yield of 4.00%, but the observed yield is 4.05%.

The innovation is +5 basis points.

If the 10-year observation has low measurement noise and level uncertainty is high, the Kalman gain can move the level factor meaningfully upward.

If the observation comes from a stale or illiquid bond encoded with high measurement noise, the filter can treat the same 5-basis-point difference as weak evidence.

Measurement noise is not “error we do not care about”

Observed yields can differ from the smooth latent curve because of:

• bid–ask spreads;
• stale or evaluated prices;
• security-specific liquidity;
• tax or repo specialness;
• bootstrapping error;
• rounding;
• microstructure noise.

Matrix R tells the model how much of that cross-sectional discrepancy should be treated as observation noise rather than a true movement in the latent curve.

Diagonal R versus correlated measurement errors

A simple model assumes measurement errors at different maturities are independent, making R diagonal.

But curve-construction errors can be correlated across neighbouring maturities. A bad instrument or interpolation issue can affect several points together.

Allowing a full R is statistically richer but adds many parameters. If the dataset cannot identify them, a complex noise matrix can simply absorb model misspecification.

Missing maturities are naturally handled

Suppose today’s 7-year quote is missing.

A Kalman implementation can omit the corresponding row of y_t and H for that update rather than inventing a fake observation.

The state prediction still exists from the transition model, and the other maturities still update it.

This is one practical advantage of state-space methods in irregular financial datasets.

Filtering versus smoothing

Filtering estimates state x_t using information available only through time t.

Smoothing estimates historical state x_t using observations that arrived after time t as well.

A Rauch–Tung–Striebel smoother can run backward after the forward Kalman pass to refine historical factor estimates.

This creates an important backtesting boundary: smoothed factors are useful for historical analysis but cannot be treated as information that was known in real time.

The likelihood comes from innovations

Under the linear-Gaussian model, the one-step innovations are Gaussian with covariance S_t.

The log-likelihood contribution is, up to constants:

−½[log|S_t| + v_t^TS_t⁻¹v_t].

Summing over time provides a likelihood for parameters such as:

• state-transition matrix A;
• process covariance Q;
• measurement covariance R;
• Nelson–Siegel decay parameter λ if not fixed.

This turns Kalman filtering into both an inference algorithm and a parameter-estimation engine.

Why λ can be difficult to identify

The Nelson–Siegel decay parameter controls where the curvature loading reaches its hump.

If maturities are sparse or concentrated in one region, many nearby λ values can produce similar fitted curves while factor estimates compensate.

That creates a flat likelihood or several local optima.

Practical implementations may fix λ from a prior calibration, profile over a sensible range, or estimate it carefully with multiple starting values and identification diagnostics.

Static NSS versus dynamic Kalman filtering

The existing NSS page owns a cross-sectional problem: find parameters that fit today’s term structure.

A dynamic state-space model adds:

• a law of motion for the factors;
• state uncertainty;
• measurement error;
• recursive updating;
• time-series forecasting.

The dynamic model can accept a slightly worse fit to one day’s noisy cross-section in exchange for a more stable and forecastable factor path.

PCA versus Kalman state-space factors

PCA diagonalizes the historical covariance matrix of yield changes and produces orthogonal directions explaining variance.

A Kalman state-space model specifies economically interpretable factor loadings and explicit factor dynamics, then filters noisy observations through time.

PCA factors are defined by a sample covariance. Kalman factors are defined by a measurement equation plus state transition.

They can look similar—often level, slope and curvature—but they are not the same algorithm or objective.

Forecasting

Once the filtered state is known, an h-step state forecast under a linear transition is obtained recursively:

x̂_t+h|t = c + A x̂_t+h−1|t.

The yield forecast is:

ŷ_t+h|t = Hx̂_t+h|t.

The forecast covariance also propagates forward through A and Q.

A point forecast without its uncertainty is only half of what the state-space model provides.

Innovation diagnostics

If the model is correctly specified, standardized innovations should behave approximately like unpredictable noise under the assumed distribution.

Diagnostics include:

• innovation means near zero;
• little serial correlation;
• covariance consistent with S_t;
• no maturity with persistent one-sided residuals;
• few extreme standardized innovations beyond what the assumed distribution permits.

Persistent residual structure says the filter is smoothing an incorrect model, not solving it.

Inputs and outputs

A dynamic yield-curve filter can require:

• yield observations and maturity mapping;
• source curve or instrument construction method;
• measurement equation and factor loadings;
• decay parameter λ;
• state-transition matrix A and intercept c;
• process covariance Q;
• measurement covariance R;
• initial state and covariance;
• missing-data policy;
• estimation and numerical-stability settings.

Outputs can include:

• filtered factors;
• smoothed historical factors;
• state covariance;
• fitted and forecast yields;
• innovation vectors and covariance;
• log-likelihood;
• forecast confidence intervals;
• parameter and residual diagnostics.

Evidence polarity: what supports confidence?

Evidence for a useful Kalman curve model includes low and structureless standardized innovations, stable factor interpretations, strong out-of-sample yield forecasts versus simple benchmarks, sensible uncertainty bands, robust parameters across nearby samples and a measurement-noise pattern consistent with market liquidity.

Evidence against confidence includes persistent maturity-specific residual bias, implausibly tiny R used to force the curve through noisy quotes, Q large enough that factors become almost random every day, λ instability, negative covariance eigenvalues from numerical failure, or forecast accuracy worse than a random-walk/no-change benchmark.

Counterexample: a smooth filtered curve can still be economically wrong

Set measurement noise very high and process noise very low.

The filter will trust its prior state strongly and ignore new market observations. The resulting curve can be beautifully smooth while lagging a genuine regime shift.

Smoothness is therefore not evidence of correctness.

Counterexample: an almost exact daily fit can destroy filtering value

Set measurement noise near zero and process noise large.

The filter can chase every cross-sectional quote movement, behaving almost like an independent daily fit.

The algorithm is still called a Kalman filter, but the dynamic prior contributes little information.

Counterexample: smoothing creates look-ahead bias

A smoothed estimate of the 2025 level factor may use yields observed in 2026.

Using that smoothed factor inside a historical 2025 trading or forecasting backtest gives the model future information.

Only filtered real-time states are valid for real-time decision simulations.

Counterexample: dynamic Nelson–Siegel is not automatically arbitrage-free

Diebold–Li-style dynamic Nelson–Siegel is primarily a statistical factor model. Its factor dynamics do not automatically enforce the cross-time no-arbitrage restrictions required by an arbitrage-free affine term-structure model.

Christensen, Diebold and Rudebusch developed arbitrage-free Nelson–Siegel classes that impose stronger finance restrictions.

This is a page-role boundary: a forecasting model can be useful without being a complete derivative-pricing model.

No-arbitrage versus statistical flexibility

Imposing no-arbitrage structure can reduce degrees of freedom and improve economic coherence for pricing.

Purely statistical state-space models can sometimes forecast well because they are flexible and focused on observed yield dynamics.

The appropriate constraint depends on the job:

• macro/curve forecasting;
• risk-factor extraction;
• relative-value monitoring;
• arbitrage-free derivative valuation.

Structural breaks and regime failure

A constant transition matrix A assumes factor dynamics are stable through time.

Monetary-policy regime changes, zero-lower-bound episodes, crises, inflation shocks or market-structure changes can alter persistence and factor volatility.

The filter can react by generating large innovations or inflating estimated process noise, but those symptoms do not prove the constant-parameter model remains adequate.

Rolling estimation, regime-switching state-space models or explicit break tests can be necessary.

Outliers and heavy-tailed measurement noise

The classical Kalman filter is optimal under linear-Gaussian assumptions.

A single bad yield quote can produce a large innovation and move the entire latent state because Gaussian likelihood still treats the observation as informative.

Robust alternatives include heavy-tailed measurement models, outlier filters, Huber-style robustification or explicit data-quality gates before the Kalman update.

Nonlinear state-space models

If factor loadings, volatility or dynamics depend nonlinearly on the state, the standard linear Kalman recursion no longer applies exactly.

Alternatives include:

• extended Kalman filter: linearize around the current state;
• unscented Kalman filter: propagate selected sigma points;
• particle filter: represent the state distribution with simulated particles.

These methods add flexibility but also computational and approximation risk.

Weak links in implementation

Maturity mismatch. A 5-year observation is accidentally mapped to a 4.5-year loading.

Unit mismatch. Percent yields and decimal yields are mixed.

Stale observations. Repeated old quotes are treated as fresh measurements.

R underestimation. Illiquid points move the entire state too strongly.

Q overfitting. Process noise absorbs every model error.

Bad initialization. Early filtered states are dominated by arbitrary starting assumptions.

Covariance instability. Floating-point update destroys symmetry or PSD.

Likelihood local optimum. Different λ/Q/R combinations fit almost equally well.

future leakage. Smoothed states enter historical forecast tests.

Diagnostics: how to test the algorithm

• synthetic-state test: simulate data from known A,Q,R and verify the filter recovers factors statistically.
• zero-noise test: with R→0 and sufficient observations, fitted yields should track the measurement equation closely.
• missing-maturity test: remove random maturities and verify the filter still updates from remaining observations.
• innovation whiteness test: inspect autocorrelation and cross-maturity residual structure.
• PSD test: require P, Q, R and S to remain valid covariance matrices.
• initialization test: vary starting factors/covariance and verify long-run estimates converge.
• λ profile test: inspect likelihood and factor stability across a grid of decay parameters.
• forecast benchmark: compare yield forecasts with random walk, static NSS and autoregressive factor models.
• filter-versus-smoother test: quantify how much historical states are revised by future data.
• regime split: estimate pre/post structural-change parameters and test stability.

What would falsify confidence?

Confidence should be withdrawn if innovations retain strong predictable structure; if covariance matrices become invalid; if parameter estimates change radically under small sample perturbations; if filtered states depend persistently on arbitrary initialization; if out-of-sample forecasts are dominated by a no-change benchmark; or if the model needs implausible Q/R values merely to fit observed curves.

Alternatives and when they are better

Static NSS is preferable when the job is one-date interpolation/extrapolation and time-series dynamics are irrelevant.

PCA is useful when the goal is empirical covariance compression and scenario factor shocks without specifying a transition model.

Affine arbitrage-free term-structure models are preferable when pricing consistency across time and maturities is central.

Regime-switching state-space models can represent discrete policy or volatility regimes.

Particle filters can handle strongly nonlinear or non-Gaussian latent-state models at greater computational cost.

How this connects to the surrounding knowledge estate

The observed yields can come from the yield-curve construction layer. Static cross-sectional shapes belong to the NSS calibration page. Historical covariance directions belong to yield-curve PCA. Filtered curve factors can feed interest-rate hedging or forecast models, but derivative pricing still needs the relevant no-arbitrage valuation framework.

Verification and update triggers

Preserve yield-source definitions, maturity mappings, λ, state-transition parameters, Q/R matrices, initialization, missing-data treatment, numerical covariance method and filtered-versus-smoothed usage. Revalidate after central-bank regime changes, benchmark transitions, yield-source migrations, persistent innovation bias, major liquidity shifts, curve-construction changes or a sustained deterioration in out-of-sample forecasts.

Primary and high-quality references

• R. E. Kalman, A New Approach to Linear Filtering and Prediction Problems, Journal of Basic Engineering, 1960.
• Francis X. Diebold and Canlin Li, Forecasting the Term Structure of Government Bond Yields, NBER Working Paper 10048; later Journal of Econometrics, 2006.
• Francis X. Diebold, Glenn D. Rudebusch and S. Boragan Aruoba, The Macroeconomy and the Yield Curve: A Dynamic Latent Factor Approach, NBER Working Paper 10616; later Journal of Econometrics, 2006.
• Jens H. E. Christensen, Francis X. Diebold and Glenn D. Rudebusch, The Affine Arbitrage-Free Class of Nelson–Siegel Term Structure Models, NBER Working Paper 13611.
• Jens H. E. Christensen, Francis X. Diebold and Glenn D. Rudebusch, An Arbitrage-Free Generalized Nelson–Siegel Term Structure Model, Federal Reserve Bank of San Francisco.

Educational boundary: This article explains state-space estimation of term-structure factors. It does not forecast rates for a reader, recommend a bond position or provide personalized financial advice.