Common failure modes in retail quant research
The same six mistakes account for most inflated backtests in independent quant research. Each one is easy to make and easy to check for.
Most inflated backtests are not the product of dishonesty. They are the product of a small, repeating set of technical mistakes, each individually easy to make and each individually capable of turning a mediocre or nonexistent edge into an impressive-looking one. Recognising these six failure modes is worth more than any single modelling technique, because a model built on top of one of them cannot be rescued by better modelling — the damage happens before the model ever sees the data.
Failure mode one: the random split disguised as validation
Standard machine-learning practice splits data into training and test sets at random. Applied to time series without modification, this is a serious mistake: a random split scatters future observations into the training set and past observations into the test set, so the model effectively trains on some of the exact conditions it is later “tested” against. The resulting test-set accuracy measures something close to memorisation, not generalisation.
Worked illustration: a dataset of 1,000 daily rows, split 80/20 at random rather than chronologically. Roughly 160 of the 200 test rows will have at least one temporally adjacent row — yesterday or tomorrow’s observation — sitting in the training set, given how densely 800 training rows are scattered across the same 1,000-row timeline. Financial data is serially correlated: a feature computed from a 20-day rolling window shares most of its underlying prices with its neighbours a few days away. Random splitting all but guarantees test rows that are highly similar to training rows the model has already seen, which is precisely why non-stationary, serially correlatedStationarity A property of a time series whose statistical characteristics — mean, variance, autocorrelation — do not change over the sample. financial data needs a chronological split, not the machine-learning default.
Failure mode two: a universe built with hindsight
Building a study on “the current S&P 500 constituents” and applying that list to historical dates silently excludes every company that was removed from the index — largely for underperforming, going bankrupt, or being acquired at a discount. The study never gets a chance to include the failures, because the failures are not in the list being tested. This is survivorship bias, and its effect compounds: because it is applied to the entire historical universe, it inflates every downstream statistic computed from that universe, not just a subset of results.
A defensible universe is built from what the index actually contained on each historical date — a point-in-time constituent list — or the bias is disclosed explicitly rather than left implicit. There is no modelling step later in the pipeline that corrects for a universe that excluded its own worst outcomes before the study began.
Failure mode three: leakage hidden inside feature preparation
Look-ahead biasLook-ahead bias Using information in a calculation that would not actually have been available at the time the decision it informs was made. does not only enter through the train/test split — it frequently enters through feature preparation performed before the split is even applied. A common version: a feature scaler (for example, one that standardises each feature to zero mean and unit variance) is fit on the entire dataset, training and test rows together, and then applied to both. The scaler’s mean and standard deviation are computed using information from the test period, which means the “test” data has already influenced how every feature was transformed before the model ever saw it.
The fix is mechanical rather than conceptual: fit any such transformation on the training rows only, and apply that same fitted transformation to the test rows without recomputing it. Inside a walk-forward validation scheme, this has to happen fresh at every fold — a scaler fit once on the full training history and reused across all folds still lets later folds’ characteristics leak backward into earlier ones through a transformation that was never refit.
A related and subtler version of the same mistake: computing a feature such as a rolling average or a rank across the full dataset before splitting, rather than computing it in a way that only ever looks backward from each row. A 20-day moving average computed naively across an entire unsplit dataset does not leak information forward by itself, but the broader pattern — compute first, split second — is exactly the habit that produces leakage the moment any feature construction step involves aggregating across the boundary, such as a cross-sectional rank computed jointly across dates that straddle the split.
Failure mode four: reporting the best of many trials as the only one
Testing many parameter combinations, feature sets, or universe definitions and reporting only the best-performing configuration produces a number that looks like a single clean result but is statistically the winner of a competition. The more configurations tried, the more likely the best of them looks strong by chance alone, independent of whether any of them found a real effect.
The correction is the deflated Sharpe ratioDeflated Sharpe ratio The Sharpe ratio adjusted for the number of configurations tried and for non-normal returns, so a result picked from many looks less impressive., which discounts a reported Sharpe ratio by the number of independent trials that produced it. Skipping this step, or — just as commonly — undercounting the actual number of trials by only logging the formal parameter grid and forgetting the informal decisions made along the way (a dropped security, an abandoned earlier version of a feature), leaves the reported number looking stronger than the evidence actually supports. Sorting a table of competing strategies by raw Sharpe ratio rather than deflated Sharpe ratio has the same effect at the reporting stage: raw Sharpe ratio systematically rewards whichever configuration happened to be searched hardest, not whichever one is most likely to be real.
Failure mode five: a backtest with no transaction costs
A backtestBacktest A simulation of a strategy's historical performance against price data it did not see during fitting, net of assumed costs and slippage. run without commission, slippage, or borrow costs measures a classification or forecasting result, not a trading strategy. This matters most for strategies that trade frequently: a signal generating a genuine 4% a year of gross edge, rebalanced weekly against a combined 7 basis points of commission and slippage per round trip, loses roughly a year to frictions alone — more than half its edge — before any of the underlying signal has been realised as return. The same signal reported without a cost assumption looks nearly twice as strong as it will ever be able to deliver.
A related version of this mistake applies costs but sets them unrealistically low, or applies them only to entries and not to exits, understating the true round-trip cost by roughly half. A transaction-cost sensitivity check — re-running the backtest at several multiples of the base assumption — reveals whether the strategy’s edge survives realistic frictions or evaporates just above the assumed baseline.
Failure mode six: no baseline, so any number looks like skill
A model reporting 61% classification accuracy, or a forecast reporting an RMSE of 0.014, means nothing without a stated comparison point. The relevant comparison for a predictive model is the naive baselineNaive baseline The simplest rule that requires no model at all — predict no change, predict the trend continues, predict the historical average — that any predictive claim must beat. — the simplest possible rule requiring no model at all, such as predicting that tomorrow looks like today. Skill relative to that baseline is ; a positive score means the model added information beyond the trivial rule, and a score near or below zero means it did not, regardless of how sophisticated the model architecture is.
The equivalent mistake in portfolio work is reporting a strategy’s absolute return without a benchmarkBenchmark A stated comparison point — an index, an equal-weight allocation, a risk-free rate — that a result must be measured against before it can be called good.. A 9%-a-year return looks respectable in isolation and looks like underperformance next to an 11%-a-year benchmark over the same period — the number by itself carries no information about whether it represents skill or merely broad market exposure.
Why these six recur together
These failure modes share a structural feature: each one makes a result look better without making the underlying research any stronger, and each one is invisible from the headline number alone. A Sharpe ratio of 1.6 looks identical whether it came from a chronological split with a proper purge gap and one pre-specified trial, or from a randomly shuffled split, a survivorship-biased universe, a scaler fit on the full dataset, the best of 80 trials, a costless backtest, and no stated baseline. Reading the method section — not just the results table — is the only way to tell which one produced the number.
Two further pieces cover the mechanics behind these failure modes in more depth: reading a backtest without fooling yourself on how to check a report for exactly these problems, and why random train/test splits break on time series on the mechanics of why a random split fails specifically on time-ordered data. None of the six failure modes above require advanced statistics to avoid — they require checking, before trusting a number, whether the split was chronological, whether the universe was point-in-time, whether any transformation touched test data early, how many trials produced the result, whether costs were realistic, and what the result was actually compared against.