Portfolio optimisation from first principles

A researcher with ten securities and a spreadsheet of daily returns has to answer one question before anything else: how much of the portfolio goes into each name. That question has a formal answer — an optimal weight vector, given a set of assumptions about expected return and risk — and an informal one, which is that the formal answer is only as good as the assumptions feeding it. This article works through both.

Mean-variance optimisation

Mean-variance optimisation asks a precise question: for a target expected return, what portfolio weights minimise variance? Formally, given expected returns $\mu$ (a vector of length $n$ ) and a covariance matrix $\Sigma$ , the problem is

\min_{w} \; w^\top \Sigma w \quad \text{subject to} \quad w^\top \mu = \mu_{\text{target}}, \quad \mathbf{1}^\top w = 1

with $w$ the weight vector. Solving this for every achievable target return traces out the efficient frontier — the set of portfolios that deliver the least variance for their level of expected return. Everything below the frontier is dominated: there exists another portfolio with the same or higher return and less risk.

A worked example. Take three assets with annualised expected returns $\mu = (8\%, 11\%, 6\%)$ and annualised volatilities $(15\%, 22\%, 9\%)$ , with pairwise correlations of 0.3 (assets 1–2), 0.1 (assets 1–3) and 0.05 (assets 2–3). The covariance matrix follows from $\Sigma_{ij} = \rho_{ij}\sigma_i\sigma_j$ . Solving the quadratic program for a target return of 9% produces a weight vector close to $w \approx (0.42, 0.21, 0.37)$ — a blend that leans on the low-volatility third asset to offset the high-volatility second one, rather than picking the two highest-return assets outright. This is the mechanical output of the optimiser: it does not know which asset “should” perform well, only how the historical covariance structure lets it diversify.

Why equal weight is the baseline, not a strawman

Every optimiser should be measured against the portfolio that requires no estimation at all: equal weight, $w_i = 1/n$ for every asset. This is not a placeholder for “no strategy” — it is a real allocation with its own risk profile, and it has a well-documented habit of matching or beating mean-variance portfolios out of sample, because it makes no bet on which estimated returns are trustworthy.

The reason is estimation error, not model error. Expected returns are the hardest quantity in finance to estimate precisely: a sample mean built on five years of monthly data carries a standard error large enough that many “different” expected returns are statistically indistinguishable from each other. Mean-variance optimisation is unusually sensitive to this — it treats small differences in $\mu$ as meaningful signal and can push weights toward extreme, concentrated positions in whichever asset happened to look best in the sample window. Equal weight sidesteps the entire problem by refusing to use $\mu$ at all.

Hierarchical risk parity

Hierarchical risk parity (HRP) is a response to a specific weakness of mean-variance optimisation: it requires inverting the covariance matrix, and that inversion becomes numerically unstable as the number of assets grows or as assets become highly correlated — small measurement errors in $\Sigma$ get amplified by the inversion into large, arbitrary weight swings.

HRP avoids inversion altogether. It works in three steps: cluster assets by correlation distance into a dendrogram (assets that move together end up nested near each other), reorder the covariance matrix according to that clustering (a step called quasi-diagonalisation), and then allocate capital recursively down the tree — splitting the current allocation between two branches in inverse proportion to each branch’s variance, rather than solving a matrix equation for the optimal split. The result is a set of weights that respects the correlation structure of the universe without ever computing $\Sigma^{-1}$ .

The practical consequence is stability. Because HRP never inverts the covariance matrix, a small change in one correlation estimate produces a small change in the resulting weights, rather than the disproportionate reweighting that mean-variance optimisation can produce from the same perturbation.

Risk parity

Risk parity targets a different objective from either mean-variance or HRP: instead of minimising variance or avoiding matrix inversion, it weights assets so that each contributes equally to total portfolio risk. The risk contribution of asset $i$ is

RC_i = w_i \cdot \frac{(\Sigma w)_i}{\sqrt{w^\top \Sigma w}}

and risk parity solves for the $w$ that sets every $RC_i$ equal. In practice this means low-volatility assets receive higher weights than high-volatility ones — the opposite instinct from a naive return-chasing allocation, and one reason risk parity portfolios are often heavier in low-volatility assets. Note that risk parity, like HRP, makes no use of expected returns at all: both are pure risk-allocation methods, which is precisely why they sidestep the estimation-error problem that afflicts mean-variance optimisation.

CVaR optimisation: minimising the tail, not the spread

Variance penalises upside and downside deviations identically, which is a strange property for a “risk” measure when investors mainly care about the downside. CVaR optimisation replaces the variance objective with the expected loss in the worst-case tail: given a confidence level (95% or 99% are typical), CVaR is the average loss conditional on the loss already exceeding the corresponding Value at Risk threshold. The optimisation problem becomes

\min_{w} \; \text{CVaR}_\alpha(w) \quad \text{subject to the same budget and return constraints}

This has a genuine advantage where return distributions are skewed or fat-tailed — a strategy with occasional large drawdowns can have modest variance but a dangerous tail, and a variance-minimising optimiser will not notice. CVaR optimisation is convex and tractable via linear programming once returns are represented by historical scenarios, which is part of why it has become a standard complement to mean-variance work rather than a replacement for it.

Estimation error: the problem underneath every method

Every method above depends on inputs estimated from a finite sample, and the finiteness matters more than the choice of method. A covariance matrix estimated from $T$ observations of $n$ assets becomes unstable as $n$ approaches $T$ — with 60 months of data and 50 assets, the sample covariance matrix is close to singular, and its inverse (needed for mean-variance and, less directly, for risk parity) becomes dominated by noise rather than signal. This is exactly why shrinkage estimators such as Ledoit-Wolf shrinkage exist: they pull the noisy sample covariance matrix toward a more stable structured target, trading a small amount of bias for a meaningful reduction in estimation variance.

The practical response is threefold: use covariance estimators built for stability rather than the raw sample matrix, treat expected-return inputs with more scepticism than expected-risk inputs (return estimates carry far larger relative error), and compare every optimised portfolio against equal weight, risk parity and HRP on the same out-of-sample data before concluding that the extra complexity earned its place. A full treatment of how to measure that out-of-sample performance — Sharpe, Sortino, drawdown and the rest — is covered in risk and return metrics that matter. What happens to a chosen allocation when the market moves sharply is covered separately in scenario and stress testing your allocation.

Key takeaways

Mean-variance optimisation minimises variance for a target return, and the resulting efficient frontier is entirely determined by the expected-return and covariance inputs supplied to it.
Equal weight is a legitimate baseline, not a placeholder: it requires no return estimate and has repeatedly matched optimised portfolios out of sample because it carries no estimation error in $\mu$ .
Hierarchical risk parity allocates down a correlation-based clustering tree specifically to avoid inverting the covariance matrix, which makes it more stable than mean-variance optimisation when correlations are high or the asset count is large.
Risk parity and CVaR optimisation solve different problems from mean-variance: risk parity equalises each asset’s contribution to total risk, and CVaR optimisation minimises expected tail loss rather than variance.
Every method here is limited by estimation error in its inputs, and comparing an optimised portfolio against equal weight and risk parity on out-of-sample data is the only honest check on whether the optimisation added value.