Benchmarks and why your comparison set matters
A return means nothing alone. What it is measured against — benchmark, naive baseline, peer universe — decides whether it is evidence of anything.
A strategy returns 14% over a year. Is that good? The question cannot be answered from the number alone — it depends entirely on what else was available during that year. If a broad equity index returned 22% over the same period, 14% represents meaningful underperformance despite being a positive, comfortable-looking number. If the index fell 18%, the same 14% represents a result most allocators would be pleased with. A return is not evidence of skill or its absence until it is placed against a benchmarkBenchmark A stated comparison point — an index, an equal-weight allocation, a risk-free rate — that a result must be measured against before it can be called good., and the choice of benchmark is itself a claim that deserves as much scrutiny as the strategy being measured.
What a benchmark actually has to be
A benchmark should be something the researcher could plausibly have held instead — investable, liquid, and constructed independently of the strategy under test. A benchmark chosen after the fact, specifically because it happened to perform badly during the test window, is not a comparison; it is a rhetorical device wearing a comparison’s clothing. The discipline that prevents this is simple to state and easy to skip under pressure: name the benchmark before running the backtest, not after seeing the result.
Beta and tracking error: how a strategy relates to its benchmark
BetaBeta A security or portfolio's sensitivity to benchmark returns, estimated by regressing its returns on the benchmark's. measures how much of a strategy’s return is explained by, and moves in proportion to, the benchmark’s return — the slope of a regression of strategy returns on benchmark returns. A strategy with a beta of 1.2 tends to move 20% more than the benchmark in either direction; a beta of 0.3 suggests the strategy’s returns are largely independent of the benchmark’s swings. Beta on its own says nothing about whether a strategy is good — a beta of 1.0 could belong to a strategy that is simply holding the benchmark, or to one with genuinely different holdings that happen to move similarly.
Tracking errorTracking error The annualised volatility of the difference between a portfolio's returns and its benchmark's, the risk term behind the information ratio. is the volatility of the difference between the strategy’s return and the benchmark’s return — how much day-to-day dispersion exists around whatever average outperformance or underperformance shows up. A strategy that beats its benchmark by 2% annually with a tracking error of 1% is a very different proposition from one that beats it by 2% with a tracking error of 15%: the first is a consistent, low-dispersion edge; the second could easily have lost by 13% in a slightly different sample period. The Information Ratio — annualised active return divided by annualised tracking error — is the metric that combines the two into a single statement of how consistently a strategy has beaten its benchmark, in the same spirit as the Sharpe ratio combining return and volatility, and is covered alongside the rest of Quant Compose’s risk-adjusted performance measures in Risk and return metrics that matter.
A worked example
Consider a strategy and a benchmark over four quarters:
| Quarter | Strategy return | Benchmark return | Active return |
|---|---|---|---|
| Q1 | 4.0% | 3.0% | +1.0% |
| Q2 | −2.0% | −5.0% | +3.0% |
| Q3 | 6.0% | 8.0% | −2.0% |
| Q4 | 1.0% | 2.0% | −1.0% |
The mean active return is per quarter. The standard deviation of the active-return column — the tracking error over this sample — is approximately per quarter. Annualising both (multiplying the mean by 4 and the standard deviation by ) gives an annualised active return of roughly and an annualised tracking error of roughly , for an Information Ratio near — a modest, unremarkable edge once its consistency is accounted for, even though three of the four quarters individually looked like wins.
The naive baseline: the comparison a model has to beat first
Before comparing a model against another model, or a strategy against a benchmark index, there is a more basic comparison that has to happen first: does the model beat doing the simplest possible thing? For a forecasting task, that simplest thing is usually “assume tomorrow looks like today” — the naive baseline. Quant Compose’s forecasting diagnostics compute a naive baseline (last observed close carried forward) identically regardless of which of the four forecasting methods is selected, and report a skill score,
A positive skill score means the model’s forecast error is smaller than simply repeating the last known value; a skill score at or below zero means the model, whatever it appears to be doing, is not adding information beyond what was already visible without it. This is a distinct question from beating a benchmark index — a forecasting model can beat the naive baseline while a strategy built on its signals still fails to beat a buy-and-hold benchmark net of costs, because the two comparisons are testing different claims. The naive-baseline comparison is covered on its own terms in Beating the naive baseline (and what it means when you don’t).
The peer universe: comparison sets beyond a single index
A single benchmark index answers “did this beat the market,” but it does not answer “was this a reasonable choice among the alternatives actually available.” A momentum strategy tested only against a broad-market index, with no comparison against equal weighting, risk parity, or a minimum-variance construction on the same universe, has only cleared the lowest bar available. Quant Compose’s portfolio construction methods include equal weight specifically because it is a baseline optimisers frequently fail to beat — a mean-variance or risk-parity result that cannot outperform simply holding every security at the same weight has not demonstrated that the added complexity earned its cost.
Choosing a comparison set before results exist
The discipline that ties all of this together is sequencing: the benchmark, the naive baseline, and the peer comparison set should be named as part of the research design, before a single result is computed, in the same way a chronological train/test split has to be decided before a model is fit rather than after seeing which split flatters the result. A comparison set chosen or adjusted after seeing performance is not a comparison; it is the same multiple-testing problem covered in Look-ahead bias and data leakage: the ten ways it happens, applied to the choice of yardstick rather than the choice of model.