Why more models is not better
The arithmetic of multiple testing: why trying more estimators, more features and more thresholds makes the best result look better and mean less.
A researcher trains one model, gets a disappointing result, and trains another. Then a third, with different features. By the fifteenth attempt, one configuration finally clears the bar the researcher set at the start. It feels like persistence paying off. It is at least partly an artefact of trying fifteen times, and the size of that artefact is not a matter of opinion — it can be calculated, and the workspace’s evidence layer calculates it rather than letting the best-looking result stand on its own.
The coin-flip intuition
Suppose fifteen configurations of a trading strategy are each, in truth, worthless — each one has exactly a 50% chance of a positive Sharpe ratio over the test window purely from sampling noise, with no genuine edge in any of them. The chance that at least one of the fifteen shows a positive result by chance alone is:
Trying fifteen worthless strategies makes it almost certain that at least one of them will look good. This is not a subtle statistical trap; it is the same logic as flipping fifteen coins and reporting the one that came up heads five times in a row while staying quiet about the other fourteen. The problem is not that any individual model’s result is calculated incorrectly — each Sharpe ratio, each ROC AUC, is arithmetically correct for the fold it was measured on. The problem is that reporting only the best of many correct numbers, without saying how many were tried, systematically overstates what has actually been found.
Why a trial ledger exists
The discipline this calls for is keeping a record of every configuration tried, not only the survivors — a trial ledger. Once every attempt is recorded, whether it involved swapping the estimator (Logistic Regression versus Random Forest versus XGBoost Classifier), changing the feature set, adjusting the classification threshold, or picking a different subset of the universe, the total count of genuine attempts becomes a known quantity rather than a guess reconstructed after the fact from memory.
A ledger recorded honestly will often be uncomfortable, because researchers rarely count the attempts they abandoned quickly, the universe they tried and dropped, or the feature set that “obviously wasn’t going to work” and was never formally scored. All of those are still trials against the same underlying question, and all of them belong in the count, because the coin-flip arithmetic above does not care whether an attempt was abandoned after five minutes or fully backtested — it cares how many independent chances the noise had to produce a good-looking number.
The deflated Sharpe ratio
The workspace’s evidence layer applies the Deflated Sharpe RatioDeflated Sharpe ratio The Sharpe ratio adjusted for the number of configurations tried and for non-normal returns, so a result picked from many looks less impressive. (Bailey and Lopez de Prado, 2014) specifically to correct for this. Instead of asking “is this Sharpe ratio positive,” it asks a sharper question: given the number of independent trials actually attempted and the length of the track record, what is the probability the true Sharpe ratio exceeds zero, once the expected maximum of many noise-only trials is subtracted out as the threshold to beat?
The intuition follows directly from the earlier arithmetic. If fifteen independent, worthless strategies are each a draw from a Sharpe-ratio distribution centred on zero, the maximum of those fifteen draws is not centred on zero — extreme-value theory says the expected maximum of many independent draws from a distribution grows with the number of draws, even when every individual draw has a mean of exactly zero. A raw Sharpe ratio of 1.2 discovered after trying two configurations is meaningfully different evidence from the same 1.2 discovered after trying two hundred, because the noise-only benchmark it needs to beat has moved. The Deflated Sharpe Ratio raises the bar in proportion to how many trials were actually run and reports the probability that the observed result clears that adjusted bar rather than the naive zero line.
This is also why the workspace’s experiment comparison tables sort candidate models by Deflated Sharpe rather than by raw Sharpe: sorting by the raw number is precisely how the single most over-searched, most fortunate configuration wins the comparison, which is the opposite of what a comparison is supposed to establish.
A worked illustration
Suppose a researcher records a trial ledger of forty attempts while searching for a profitable classification-based strategy, and the best of the forty shows a raw annualised Sharpe ratio of 1.4 over a two-year out-of-sample test window. Two competing claims are possible:
- “This strategy has a Sharpe ratio of 1.4” — treating the number as if it were the only thing ever tried.
- “The best of forty tried configurations has a Sharpe ratio of 1.4” — the honest version, which is exactly what a Deflated Sharpe Ratio calculation is built to evaluate.
Under the second framing, the probability that this specific result reflects genuine skill rather than the best draw among forty noise-only attempts is computed directly from the trial count, the track-record length, and the higher moments (skewness and kurtosis) of the strategy’s own return distribution — not asserted, and not something a researcher can improve by simply not mentioning how many configurations were tried. A track record too short, or a trial count too high relative to that track record, will drive the deflated probability toward the region where the honest verdict is “insufficient evidence,” even while the raw 1.4 sits on the page looking impressive. The minimum track-record length calculation answers the related question directly: given the observed Sharpe ratio and its variance, how long would the record need to run before that Sharpe ratio is statistically distinguishable from zero at all — often a materially longer window than the one actually available.
What this means for practice, not just theory
None of this argues against trying multiple models — comparing Logistic Regression against Random Forest against XGBoost Classifier on identical data, an identical split and an identical threshold is exactly what the Experiment Comparison stage of Tune & Experiment is for, and it is legitimate research. The discipline is in what happens after the comparison: reporting the trial count alongside the winner, letting the Deflated Sharpe Ratio or an equivalent multiple-testing correction stand next to the raw number rather than replacing it, and treating a large gap between the raw and deflated figures as informative rather than as noise to explain away.
The same arithmetic that governs strategy search governs model and feature search, hyperparameter search, and even universe selection — trying twenty candidate securities and reporting results only for the five that worked is the identical failure mode with a different name, which is why the workspace’s Universe Selection Trials method exists to count that particular kind of search as well. The topic of controlling the false-discovery rate across hypothesis tests, and the related overfitting-detection methodology used once a single model is chosen, is covered fully in overfitting: detection, not prevention and in beating the naive baseline.