Reading a backtest without fooling yourself
A backtest report has a correct reading order. Read the worst fold and the cost sensitivity before the headline Sharpe ratio, not after.
A report lands on a desk: annualised return 32%, Sharpe ratio 2.1, maximum drawdown 11%. Read in that order, it looks like a strategy worth funding. Read in the right order — worst fold first, cost sensitivity second, headline number last — the same three numbers can tell a very different story. Knowing which order to read a backtestBacktest A simulation of a strategy's historical performance against price data it did not see during fitting, net of assumed costs and slippage. report in, and which numbers are load-bearing versus decorative, is a separate skill from building the backtest itself.
Question one: in-sample, or out-of-sample?
Before any other number matters, establish whether the reported statistics were computed on data the model was fitted to, or on data it never saw during fitting. An in-sample Sharpe ratio of 2.1 says almost nothing about future performance — it describes how well the model fit its own training data, which any sufficiently flexible model can do well given enough parameters. Only an out-of-sample figure, computed on data the model’s parameters were not chosen to fit, carries information about whether the strategy has found something that generalises.
A report that states a single performance number without saying which kind it is should be treated as in-sample until proven otherwise. This is not paranoia; it is the default a reader should hold, because in-sample results are considerably easier to produce and far more common in informal research.
The average fold is not the headline — the worst one is
Serious out-of-sample testing uses walk-forward validationWalk-forward validation A sequence of out-of-sample test periods with refitting at each origin, so every reported result is on data the model never trained on.: a sequence of out-of-sample periods, with the model refit at each origin point using only data available up to that point. Because performance varies across market conditions, a walk-forward report aggregates results across every fold with four summary statistics — mean, standard deviation, minimum, and maximum — rather than a single average.
The instinct is to read the mean and move on. The more useful number is the minimum — the worst fold. Consider a five-fold walk-forward test of a classification model, reporting ROC AUCROC AUC The area under the receiver operating characteristic curve; the probability a classifier ranks a random positive case above a random negative one. per fold:
| Fold | ROC AUC |
|---|---|
| 1 | 0.64 |
| 2 | 0.61 |
| 3 | 0.59 |
| 4 | 0.66 |
| 5 | 0.51 |
Mean ROC AUC across the five folds is 0.602 — comfortably above the 0.50 line that represents no discriminative power. But fold 5, at 0.51, is barely distinguishable from a coin flip. A model whose worst fold is this close to random is a model that failed to generalise in at least one market regime the walk-forward split happened to isolate — and there is no guarantee the next live period will resemble folds 1 through 4 rather than fold 5. The mean describes the average case; the worst fold describes the case a strategy actually has to survive to avoid a bad year.
Degradation: how much of the in-sample result survived
A useful single number for judging whether a strategy is likely to disappoint live is the degradation statistic: the ratio of out-of-sample Sharpe ratio to in-sample Sharpe ratio, alongside the share of out-of-sample folds that were outright negative. As a worked example, a strategy with an in-sample Sharpe ratio of 2.4 and an out-of-sample Sharpe ratio of 1.8 has retained of its in-sample performance — a healthy retention rate, consistent with a real, if modest, effect surviving contact with unseen data. The same in-sample Sharpe ratio of 2.4 paired with an out-of-sample Sharpe ratio of 0.6 retains only 25% — a strong signal that most of the in-sample number was the model fitting noise specific to the training window, not a pattern that persists.
Retention above roughly 70%, with fewer than a quarter of folds negative, describes a result behaving the way a genuine effect should. Retention below roughly 40% describes a result that has mostly evaporated out of sample, whatever the in-sample headline claimed. This single ratio is a faster diagnostic than staring at the in-sample and out-of-sample numbers side by side and trying to judge the gap by eye.
Costs turn a signal into a strategy — or erase it
A classification result with strong ROC AUC does not automatically translate into a profitable strategy, because classification accuracy and financial value answer different questions. The bridge between them is the backtest’s cost model: commission charged on every executed change in exposure, slippage on every fill, and — for short positions — a borrow cost accrued per period.
Worked example: a strategy that rebalances weekly, with a 5-basis-point commission and a 2-basis-point slippage assumption charged on each round-trip change in exposure, pays roughly a year in frictions before any of its signal has contributed a cent of return. A signal generating a genuine but modest 4% a year of gross edge is roughly halved by that cost load. The same signal traded monthly instead of weekly — a year — keeps most of its edge intact. Trading frequency is not a free parameter; it interacts directly with whether a real signal survives its own implementation.
A transaction-cost sensitivity check — re-running the same backtest across a range of cost multiples — answers a sharper question than a single cost assumption can: at what cost level does the strategy’s edge disappear entirely? A strategy whose Sharpe ratio collapses to zero at 1.5× the assumed cost is far more fragile than one that stays positive up to 4× the assumed cost, even if both look identical under the base-case assumption.
Drawdown and the benchmark, not just the average return
Average annual return understates the experience of holding a strategy through its worst period. DrawdownDrawdown The decline from a running peak in cumulative value; maximum drawdown is the worst peak-to-trough loss observed in a series. — the peak-to-trough decline before a new high is reached — captures that worst period directly, and its duration matters as much as its depth: a 15% drawdown that resolves in two months is a different experience from a 15% drawdown that persists for two years, even though both share the same maximum figure.
A return figure also means little without its comparison point. A strategy returning 9% a year against a benchmarkBenchmark A stated comparison point — an index, an equal-weight allocation, a risk-free rate — that a result must be measured against before it can be called good. that itself returned 11% a year has underperformed, despite the positive absolute number. Reading a backtest means checking the benchmark line on the same chart, not just the strategy’s own equity curve in isolation, and checking whether the strategy’s risk — its value at riskValue at risk The loss threshold a portfolio is expected to exceed with a stated probability over a stated horizon, reported as a positive number. or its volatility — was actually lower than the benchmark’s, since a strategy that merely takes on more risk to generate more return has not demonstrated skill, only leverage.
A reading order that resists the headline
Put together, a defensible reading order for any backtest report is: confirm out-of-sample status first; read the worst fold, not the mean; check the degradation ratio between in-sample and out-of-sample performance; check the cost sensitivity and at what multiple the edge disappears; then, and only then, look at the headline return and Sharpe ratio, now with context for what they actually represent. Undefined or unavailable statistics in a report should appear stated as missing, not silently replaced with a zero — a report that shows 0.0 for “not computed” is arguably more dangerous than one that shows nothing at all, because a zero looks like an answer.
None of these checks require distrust of the researcher who produced the report. They require treating a backtest the way any other quantitative claim should be treated: as a number that has to show its working, not a number to be taken on the strength of how good it looks on first read. Two related pieces cover the mechanics behind these checks in more depth — signal versus noise on what a result has to survive before it counts as evidence, and walk-forward validation, fold by fold on how a walk-forward split is actually constructed fold by fold.