Signal versus noise: what a result has to survive

A researcher tests 40 variations of a mean-reversion signal — different lookback windows, different entry and exit thresholds, a handful of universe choices — and finds one configuration with a Sharpe ratioSharpe ratio Annualised excess return over annualised volatility, the standard reward-per-unit-of-risk statistic for a strategy or portfolio. of 1.8 over a five-year backtest. That is a genuinely good-looking number. It is also, on its own, close to meaningless, because the researcher has not yet answered the question that determines whether 1.8 reflects a real effect or the best outcome of 40 rolls of a noisy die.

Distinguishing signal from noise is not about running a fancier model. It is about knowing what a number has to survive before it earns the word “result,” and reporting the parts of the process that make survival checkable.

Noise looks like a result more often than intuition suggests

Financial returns are noisy: daily and weekly moves contain far more randomness than structure, especially at the level of an individual signal rather than a broad market factor. That noise has a predictable statistical consequence once more than one configuration is tried. If 40 genuinely random, equally noisy strategies are backtested against data with no real edge in it, ordinary sampling variation guarantees that some of them will show a strong positive Sharpe ratio purely by chance — not because any of them found something, but because with 40 independent draws from a noisy distribution, the maximum of the batch drifts well above the average of the batch. The more configurations tried, the further that maximum drifts, and the more likely it is to look, by itself, like a real discovery.

This is the multiple-comparisons problem, and it is the single most underappreciated source of false results in retail and academic finance alike. It does not require anyone to act dishonestly. A researcher who tries 40 reasonable variations of an idea, in good faith, and reports only the best one, has still built a report that overstates the strength of the evidence — because the report is silent about the other 39.

Correcting for how many things were tried

The deflated Sharpe ratioDeflated Sharpe ratio The Sharpe ratio adjusted for the number of configurations tried and for non-normal returns, so a result picked from many looks less impressive. exists specifically to answer this: given the reported Sharpe ratio, the number of independent trials that produced it, and the non-normality of the return distribution, what is the probability that a Sharpe ratio this high would have appeared by chance alone, from a strategy with no real edge?

A simplified illustration of the mechanism: as more independent trials are run against noise, the expected maximum Sharpe ratio among them rises even though the true Sharpe ratio of every one of them is zero. Roughly, the expected best-of-batch Sharpe ratio grows with the number of trials as

$\mathbb{E}[\max \text{SR}] \approx \sigma_{SR}\sqrt{2\ln N}$

where $N$ is the trial count and $\sigma_{SR}$ is the standard deviation of Sharpe ratios across trials from sampling noise alone. Doubling the number of configurations tried from 10 to 20 does not double the expected best-of-batch Sharpe ratio, but it reliably pushes it upward — meaning the same underlying “no real edge” scenario produces a better-looking best result the more configurations are searched. A deflated Sharpe ratio takes this into account and asks whether the reported result clears that inflated bar, not whether it clears zero.

This is why an honestly reported research process states the trial count alongside the headline number. A Sharpe ratio of 1.8 from the only configuration tried and a Sharpe ratio of 1.8 from the best of 40 tried are different claims, even though the number on the page is identical.

The trial count is bigger than it looks

Researchers frequently undercount their own trials because they only count the parameter grid they formally searched, and forget the decisions made before that grid was ever built. Dropping a security from the universe because “it didn’t fit,” discarding an earlier version of a feature because a later one “worked better,” or quietly restarting after an early version of the model performed poorly — all of these are trials. They shaped the final configuration exactly as much as a documented grid search did, even though no formal record of them exists. A trial ledger that only logs the parameter sweep and misses the informal decisions before it will understate the deflation the result actually needs, sometimes substantially.

How long a track record has to be before it means anything

A related and often-skipped question: even a genuinely honest Sharpe ratio, from a single pre-specified test, needs enough history behind it before its precision is worth trusting. A Sharpe ratio of 1.5 computed from six months of daily returns and a Sharpe ratio of 1.5 computed from five years of daily returns are not equally credible, because the shorter series carries far more sampling uncertainty around that estimate. The concept of a minimum track-record length formalises this: given an observed Sharpe ratio and its estimation uncertainty, how long would the track record need to be before that Sharpe ratio is statistically distinguishable from a lower benchmark, such as zero or the risk-free rate? A striking headline number from a short history is frequently a number that has not yet earned the confidence its size implies.

Worked example: the same headline, two different stories

Consider two reports, both stating “annualised Sharpe ratio of 1.6, five-year backtest, monthly rebalancing.”

Report A: one strategy specification, decided before the data was examined, tested once. No parameter search. Five years of monthly observations (60 data points) is a modest sample, so the report also states the estimated standard error around the Sharpe ratio and notes that a longer track record would be needed to distinguish 1.6 confidently from, say, 1.0.

Report B: the best of 85 parameter combinations tried across three universe definitions, with the deflated Sharpe ratio not computed or not disclosed.

Both reports show 1.6. Report A is a modest, honestly bounded claim. Report B is very likely to be substantially weaker than it looks once deflated for 255 effective trials (85 combinations × 3 universes) — and without that deflation, a reader has no way to tell the two apart from the headline alone.

What else a result has to survive

Trial count and track-record length are two of several things a claimed result has to clear before it is evidence rather than noise. A result also has to survive being tested on data the model never trained on — see reading a backtest without fooling yourself for how to read an out-of-sample report rather than an in-sample one. And it has to survive scrutiny of how the train/test boundary itself was drawn: a result that looks strong only because look-ahead biasLook-ahead bias Using information in a calculation that would not actually have been available at the time the decision it informs was made. let future information leak into training is not a result at all, regardless of how many trials were run or how long the track record is. Purging and embargoing explained visually covers the specific mechanics that keep that boundary honest.

None of this means a good result is rare or that quantitative research is futile. It means the burden of proof is specific and checkable: name the trial count, name the comparison, state the sample length, and show the result survives a train/test boundary that could not have leaked. A number that clears all four is a genuinely different thing from a number that clears none of them, even when both numbers are 1.6.