Reader question: If we know the expected return and risk relationships among several assets, how does a computer choose portfolio weights that minimise risk for a target return — and why can the mathematically optimal answer still be a poor real-world forecast?
Mean–variance optimisation begins with two objects:
- a vector of expected returns μ;
- a covariance matrix Σ.
For portfolio weights w, expected portfolio return is:
E[Rp] = μᵀw.
Portfolio variance is:
Var(Rp) = wᵀΣw.
The algorithm chooses w subject to constraints such as weights summing to one, minimum or maximum holdings, no shorting, turnover limits or a required expected return. When the covariance matrix is positive semidefinite and the constraints are convex, the problem is a convex quadratic program.
What this page owns — and what it does not
This page owns the computational transformation:
expected-return vector + covariance matrix + constraints → portfolio weights on an efficient frontier.
It does not replace bond-index construction, which defines benchmark membership and weights; VaR and expected-shortfall algorithms, which measure portfolio loss distributions; or bond immunisation, which targets asset–liability interest-rate matching.
This is optimisation and linear-algebra education. It does not recommend any portfolio, asset allocation or investment strategy.
Markowitz’s core insight: covariance matters
Harry Markowitz’s 1952 paper Portfolio Selection formalised a central idea: asset risk cannot be assessed only one security at a time. What matters for portfolio risk is how assets move together.
Two individually volatile assets can form a lower-volatility portfolio if their returns are imperfectly correlated.
The Nobel Prize summary of Markowitz’s work describes his contribution as a foundation for analysing portfolio choice in terms of expected return and risk, with diversification arising from relationships among assets rather than isolated security characteristics.
Build the expected-return vector
For n assets:
μ = [μ₁, μ₂, …, μₙ]ᵀ.
If the weights are:
w = [w₁, w₂, …, wₙ]ᵀ,
then expected portfolio return is linear:
μᵀw = Σ wᵢμᵢ.
This is easy to calculate. The difficult part is that expected returns are not observed constants. They must be estimated, forecast or specified.
Build the covariance matrix
The covariance matrix is:
Σ = [σij].
Diagonal entries are variances:
σii = Var(Rᵢ).
Off-diagonal entries are covariances:
σij = Cov(Rᵢ, Rⱼ).
For two assets:
Var(Rp) = w₁²σ₁² + w₂²σ₂² + 2w₁w₂Cov(R₁,R₂).
The cross-term is where diversification enters mathematically.
Correlation and covariance
Covariance can be written using correlation:
Cov(Rᵢ,Rⱼ) = ρijσᵢσⱼ.
If correlation is less than +1, combining assets can reduce variance below a simple weighted average of individual variances.
If correlation is −1 in a two-asset toy case and volatilities/weights are matched appropriately, variance can theoretically be driven to zero. Real asset correlations are unstable and rarely offer such perfect cancellation.
The global minimum-variance problem
The simplest Markowitz problem is:
minimise wᵀΣw
subject to:
1ᵀw = 1.
If short selling is allowed and Σ is invertible, the closed-form solution is:
wGMV = Σ⁻¹1 / (1ᵀΣ⁻¹1).
This formula is elegant but fragile. If the covariance matrix is nearly singular, small input changes can produce very large changes in Σ⁻¹ and therefore in the weights.
Target-return formulation
To find the minimum-variance portfolio with a chosen expected return r*:
minimise wᵀΣw
subject to:
1ᵀw = 1,
μᵀw = r*.
Varying r* traces the efficient frontier: the set of portfolios with the smallest variance achievable for each expected return.
Why the frontier is not “the best portfolio”
The efficient frontier contains many portfolios. It does not select one universal optimum because different objectives imply different trade-offs between expected return and variance.
A risk-aversion formulation can instead maximise:
μᵀw − λ wᵀΣw,
where λ > 0 controls the penalty on variance.
Large λ pushes toward lower variance. Small λ gives expected return more influence.
Quadratic programming structure
A convex quadratic program has a quadratic objective and linear equality/inequality constraints.
A standard portfolio problem can be written:
minimise ½wᵀQw + cᵀw
subject to:
Aw = b,
Gw ≤ h.
For mean–variance optimisation, Q is derived from the covariance matrix and c from expected-return or cost terms.
Convexity matters because a positive-semidefinite Q gives one global optimum rather than a landscape full of local minima.
Positive semidefinite covariance is a mathematical invariant
For any vector x:
xᵀΣx ≥ 0.
A true covariance matrix must be positive semidefinite because a variance cannot be negative.
If a matrix assembled from inconsistent pairwise correlations has a negative eigenvalue, the optimiser may encounter a non-convex or nonsensical risk surface.
This is a high-value data-quality diagnostic.
Why sample covariance can be unstable
Suppose there are 100 assets but only 120 monthly observations. The covariance matrix contains 5,050 unique variance/covariance terms. Estimating that many relationships from a short history creates substantial noise.
Highly correlated assets can also make the matrix poorly conditioned.
The optimizer then “sees” tiny estimated differences as exploitable certainty and can produce extreme offsetting long and short positions.
Condition number as a warning signal
The covariance matrix eigenvalues measure risk along orthogonal directions.
If the largest eigenvalue is enormous relative to the smallest positive eigenvalue, the matrix has a high condition number and inversion is numerically sensitive.
A portfolio engine should therefore inspect:
- minimum eigenvalue;
- condition number;
- effective rank;
- pairwise correlations;
- sample length relative to asset count.
Shrinkage: trade noise for bias
A common stabilisation method is covariance shrinkage:
Σshrunk = αF + (1−α)Σsample,
where F is a structured target such as a diagonal, constant-correlation or factor covariance matrix.
Shrinkage deliberately adds bias to reduce estimation variance.
The goal is not to recover the “true” covariance exactly. It is to produce a more stable risk estimate for the optimisation task.
Factor covariance models
Instead of estimating every pairwise covariance directly, returns can be represented by a smaller set of common factors:
R = Bf + ε.
Then:
Σ ≈ BΣfBᵀ + D,
where D represents idiosyncratic variance.
Factor models reduce dimensionality and can improve conditioning, but they introduce model risk through factor choice and exposure estimation.
Real constraints change the solution dramatically
Common convex constraints include:
- budget: 1ᵀw = 1;
- long-only: wᵢ ≥ 0;
- position cap: wᵢ ≤ uᵢ;
- sector exposure: lower ≤ Asectorw ≤ upper;
- tracking exposure: bounds relative to a benchmark;
- turnover: Σ|wᵢ − wᵢold| ≤ τ, when represented with convex auxiliaries;
- factor neutrality: Bᵀw = target.
Stanford work by Stephen Boyd and collaborators shows how practical portfolio construction combines mean–variance structure with constraints and transaction costs in convex optimisation frameworks.
Why a no-shorting constraint can improve stability
Without constraints, an optimiser may take +600% in one asset and −550% in another because their estimated returns/covariances differ slightly.
A long-only constraint:
wᵢ ≥ 0
removes those leveraged cancellation trades.
This can reduce mathematical efficiency under the estimated model while improving robustness to estimation error.
Constraints can make the problem infeasible
Suppose the user demands:
- weights sum to 100%;
- every asset weight ≤ 5%;
- only 10 assets may be held.
Maximum possible invested weight is only 50%. The constraints are inconsistent.
A good optimiser returns infeasible with diagnostic information. It does not quietly relax rules unless a defined hierarchy permits it.
KKT conditions and active constraints
For convex problems, Karush–Kuhn–Tucker conditions characterise optimality under regularity assumptions.
Operationally, the interesting output is not only the optimal weights but also which constraints are binding and their dual values.
A binding 10% sector cap can explain why the optimiser does not allocate more to assets that otherwise look attractive under the estimated mean/covariance inputs.
Transaction costs turn allocation into a trading problem
A target portfolio is not enough if moving from today’s holdings is expensive.
A convex objective can include a penalty:
Expected utility − transaction-cost penalty.
Linear or piecewise-linear cost approximations can preserve convexity. Fixed fees, integer share requirements and minimum trade sizes can introduce non-convexity or mixed-integer decisions.
Boyd and collaborators have published portfolio-construction methods explicitly addressing transaction costs and more complex non-convex terms.
Expected-return estimation is usually the weakest input
Covariance can be noisy. Expected returns are often even harder to forecast.
A small change in μ can move the optimal portfolio sharply because the optimiser treats the estimate as a genuine expected-return advantage unless constrained or regularised.
This is one of the central reasons mathematically optimal portfolios can be economically unstable.
Optimisation amplifies input differences
The optimizer is designed to exploit differences. If Asset A has estimated return 7.01% and Asset B 7.00%, the solver may act on that one-basis-point difference even when the forecast uncertainty is several percentage points.
Optimisation therefore converts estimation noise into allocation certainty unless the model explicitly represents uncertainty or uses robust constraints.
Evidence polarity: what supports confidence?
Evidence for an optimisation result includes a positive-semidefinite and well-conditioned risk model, stable weights under small input perturbations, transparent active constraints, out-of-sample risk close to predicted risk, reasonable turnover, and similar solutions under alternative defensible covariance estimates.
Evidence against confidence includes huge long/short offsets, high sensitivity to one data point, large weight changes from tiny expected-return revisions, near-singular covariance, constraint infeasibility, or out-of-sample variance far above the model estimate.
Counterexample: mathematical optimality does not prove forecast accuracy
A quadratic solver can find the exact global minimum for the numbers it was given.
If expected returns are wrong and covariances shift, the exact optimum of the estimated model can be poor in reality.
This is a fundamental falsifier:
solver correctness ≠ model-input correctness.
Counterexample: diversification does not mean equal weighting
Equal weighting can be diversified by count, but it ignores covariance and differing volatility.
Mean–variance diversification asks how each position changes total portfolio variance at the margin.
A small allocation to a low-correlation asset can reduce risk more than a larger allocation to another highly correlated asset.
Counterexample: low historical correlation can disappear in stress
Two assets may show modest correlation during normal markets and move together during crises.
A covariance matrix estimated from calm periods can therefore overstate diversification precisely when risk matters most.
Scenario and stress analysis remain necessary even if the mean–variance optimizer is mathematically correct.
Counterexample: more assets can make estimation worse
Adding assets expands opportunity but also expands the number of covariance terms and expected returns to estimate.
With fixed historical data, a larger universe can make the optimisation problem statistically less reliable.
More choice is not automatically more information.
Robust optimisation
Robust methods replace one exact estimate with an uncertainty set. Instead of assuming μ and Σ are known perfectly, they search for weights that perform acceptably under a range of plausible inputs.
This can reduce extreme allocations and improve stability but typically sacrifices some in-sample efficient-frontier performance.
Regularisation
Adding a penalty such as:
γ||w||²
can discourage concentrated or leveraged weights.
An L1-style penalty can encourage sparsity under suitable formulations. Turnover penalties can keep new portfolios closer to existing holdings.
Regularisation is another way of admitting that the estimated optimum is not certain.
Inputs and outputs
A robust mean–variance engine can require:
- asset universe and identifiers;
- expected-return estimates;
- covariance/risk model;
- current holdings;
- budget and leverage rules;
- shorting permissions;
- position/sector/factor constraints;
- transaction-cost estimates;
- benchmark exposures if relevant;
- risk-aversion or target-return parameter;
- solver and numerical tolerances.
Outputs can include optimal weights, expected return, predicted variance/volatility, active constraints, dual values, turnover, transaction-cost estimate, efficient-frontier points, solver status and sensitivity diagnostics.
Efficient-frontier construction
One algorithm for tracing the frontier is:
- find the minimum feasible expected return and maximum relevant target range;
- choose a grid of target returns;
- solve the minimum-variance quadratic program at each target;
- discard infeasible targets;
- plot expected return against portfolio volatility.
The result is a numerical approximation of the feasible efficient frontier under the chosen constraints.
Frontier diagnostics
A suspicious frontier can reveal problems:
- non-monotonic risk along an allegedly efficient branch;
- jumps caused by solver failures;
- isolated impossible points;
- extreme weights appearing suddenly at tiny target-return changes;
- frontier changes drastically under a one-day data shift.
These are not necessarily bugs, especially when constraints become active, but they require explanation.
Backtesting without look-ahead
To test an optimiser fairly:
- estimate μ and Σ using only information available at each rebalance date;
- solve the portfolio;
- apply it to later, unseen returns;
- include turnover and transaction costs where the use case requires them;
- repeat through time.
Using future returns in the covariance or expected-return estimate creates look-ahead bias.
Predicted versus realised variance
At each rebalance, record predicted variance:
wᵀΣw.
Then compare it with realised variance over the subsequent horizon.
Persistent underprediction is evidence that the risk model, regime assumptions or rebalance horizon is inadequate.
Weak links in implementation
Return alignment error. Asset histories use inconsistent dates or currencies.
Missing-data imputation. Correlations are created by filling gaps badly.
Non-PSD covariance. Pairwise estimates form an invalid risk matrix.
Near singularity. Inversion magnifies noise.
Look-ahead bias. Future information enters estimates.
Survivorship bias. Failed or delisted assets vanish from historical data.
Unit mismatch. Daily expected returns are combined with annual covariance or vice versa.
Constraint drift. Legal or portfolio limits change but the optimiser keeps the old matrix.
Ignoring costs. Tiny theoretical gains create huge turnover.
Diagnostics: how to test the engine
- weight-sum test: verify budget equality within tolerance.
- constraint test: independently check every bound and exposure after solving.
- PSD test: inspect covariance eigenvalues.
- condition-number test: flag near-singular risk matrices.
- closed-form test: compare unconstrained/global-minimum-variance solutions with the analytic formula.
- perturbation test: add small noise to expected returns/covariance and measure weight stability.
- alternative-risk-model test: compare sample covariance, shrinkage and factor covariance.
- out-of-sample variance test: compare predicted and realised risk.
- cost-aware test: verify whether expected benefit survives transaction costs.
- infeasibility test: feed contradictory constraints and require an explicit infeasible status.
- reproducibility test: same frozen inputs and solver tolerances should reproduce the solution.
What would falsify confidence?
Confidence should be withdrawn if the covariance matrix is invalid or unstable; tiny perturbations produce radically different allocations; the solution violates independent constraint checks; out-of-sample risk repeatedly exceeds predictions; transaction costs erase the claimed advantage; the solver reports optimality on inconsistent units; or the result depends on future data that were unavailable at the decision time.
Alternatives
Risk-parity methods focus on risk contributions rather than expected-return forecasts. Minimum-variance portfolios avoid explicit return estimates. Black–Litterman blends equilibrium returns with views. Robust optimisation models parameter uncertainty. CVaR/expected-shortfall optimisation targets tail-loss measures rather than variance. Scenario and stochastic-programming methods use explicit future states. Equal-weighting is a simple benchmark that avoids estimation-heavy optimisation.
Mean–variance optimisation remains foundational because it turns diversification into a precise quadratic problem. Its main lesson is not that a solver can discover one permanently optimal portfolio. It is that covariance, constraints and estimation error can all be made explicit and tested.
How this connects to the surrounding knowledge estate
If the universe is a benchmark, portfolio constraints can be expressed relative to index weights. Portfolio risk can be challenged using VaR, expected shortfall and stress scenarios. Fixed-income portfolios can add duration/key-rate immunisation constraints. Changes in covariance behaviour over time belong in model-drift monitoring.
Verification and update triggers
Preserve the asset universe, return-estimation window, expected-return method, covariance-model version, annualisation convention, constraints, transaction-cost model, benchmark, solver version and tolerances. Revalidate after universe changes, covariance regime shifts, major volatility events, benchmark reconstitutions, constraint changes, data-vendor migrations, solver upgrades or repeated out-of-sample risk misses.
Primary and high-quality references
- Harry Markowitz, Portfolio Selection, The Journal of Finance, March 1952.
- Nobel Prize, Harry M. Markowitz — Facts, summarising the portfolio-selection contribution recognised in the 1990 Economic Sciences Prize.
- Stephen Boyd and collaborators, Portfolio Construction as Linearly Constrained Separable Optimization, Optimization and Engineering, 2023.
- Stephen Boyd, Miguel Lobo and Maryam Fazel, Portfolio Optimization with Linear and Fixed Transaction Costs.
- CVXPY, convex optimisation software and documentation, for independently reproducible quadratic-program implementations.
Educational boundary: This article explains quadratic optimisation and estimation risk. It does not recommend any portfolio or asset allocation and is not personalized financial advice.