Beating the naive baseline (and what it means when you don't)
Why every forecast has to be measured against last-value-carried-forward first, and how to read a skill score that comes out negative.
A forecasting model reports an RMSE of 0.014 on next-day returns. That number is meaningless on its own — RMSE has no natural scale that says “good” or “bad” independent of the series being forecast. The only question worth asking is: 0.014 compared to what? The answer the workspace uses is a specific, deliberately unglamorous one — a model that simply carries the last observed value forward, unchanged.
What the naive baseline is
The naive baselineNaive baseline The simplest rule that requires no model at all — predict no change, predict the trend continues, predict the historical average — that any predictive claim must beat. forecasts tomorrow’s value as today’s value. For a return series, it forecasts zero return every day. For a price series, it forecasts tomorrow’s close equal to today’s close. It uses no model, no features, no fitted parameters — it is the forecast produced by someone with the data but no analytical method at all.
This sounds like it should be trivially easy to beat. It frequently is not. Financial return series are, to a first approximation, close to unpredictable at short horizons — a point covered in returns, log returns and stationarity — which means the naive baseline’s forecast error is already close to the practical floor a model can reach. Beating it by a wide margin on a genuinely liquid, well-arbitraged instrument should be treated as a signal to check the pipeline for leakage before treating it as a discovery.
The skill score
Quant Compose’s forecast diagnostics compute out-of-sample accuracy against the naive baseline, computed identically regardless of which of the four supported forecasting methods — linear lag model, naive itself, random forest lags, or XGBoost lags — is selected. The comparison is expressed as a skill score:
A skill score of 0 means the model performs identically to doing nothing. A positive skill score means the model’s forecast error is smaller than the naive baseline’s — it is adding information. A negative skill score means the model is worse than simply carrying the last value forward, which is a stronger and more useful statement than “the model’s RMSE was 0.014,” because it puts that number in the only context that determines whether the model is worth using at all.
A worked example
Suppose the naive baseline achieves an RMSE of 0.0180 on a held-out test period, and a random-forest-lags model achieves an RMSE of 0.0158 on the same period.
A skill score of 0.122 means the model’s forecast error is about 12.2% smaller than simply carrying the last value forward. That is a real, if modest, improvement — the kind of number a careful researcher reports precisely, without inflating it into a claim of “high accuracy.”
Now suppose a second model, a more heavily parameterised gradient-boosted lag model, achieves an in-sample RMSE far below either of these — but an out-of-sample RMSE of 0.0201.
A skill score of −0.117 means this model is worse than doing nothing on the exact period it was meant to forecast. Its in-sample performance may have looked considerably better than the first model’s — that is precisely the situation this metric exists to catch. Reporting only the in-sample RMSE, or even only the out-of-sample RMSE without the naive comparison, would let this model appear competitive when it is, on the one comparison that matters, actively harmful.
| Model | RMSE (OOS) | Skill vs naive |
|---|---|---|
| Naive (last value) | 0.0180 | 0.000 |
| Random forest lags | 0.0158 | +0.122 |
| Gradient-boosted lags (overfit) | 0.0201 | −0.117 |
Why “beats the baseline” is a low bar, and still an essential one
Beating the naive baseline is not evidence that a model is good. It is evidence that a model is not actively worse than the simplest possible forecast — a necessary condition, not a sufficient one. A model with a skill score of +0.03 has cleared the bar, but +0.03 out of sample, especially before accounting for the number of configurations tried to find it, may not survive the multiple-testing corrections covered in overfitting: detection, not prevention. The naive baseline check and the overfitting corrections are sequential filters, not substitutes for each other — a model has to pass both.
Naive baselines belong in classification comparisons too
The same discipline applies outside forecasting proper. In classification, the equivalent naive comparison is a model that predicts the majority class every time, or one that predicts based on unconditional class frequency alone. A classifier reporting 0.58 ROC AUCROC AUC The area under the receiver operating characteristic curve; the probability a classifier ranks a random positive case above a random negative one. is meaningfully different from a classifier reporting the same 0.58 on a target where the base rate itself already produces something close to 0.5 by chance — the interpretation of ROC AUC relative to a baseline of 0.5, and the importance of the worst fold alongside the mean, is covered in ROC AUC, worst-fold performance, and reporting honestly. The underlying principle is the same one this article makes for forecasting: a raw error metric or a raw accuracy figure means little until it is placed next to what a method with no analytical content at all would have achieved on the identical data.
The baseline has to be computed on the same fold, every fold
Skill score is only meaningful when the naive baseline is computed on the identical out-of-sample window as the candidate model — never on a separate, more favourable stretch of history, and never pooled across the whole dataset while the model’s error is measured fold by fold. Consider a five-fold walk-forward run where the naive baseline’s RMSE varies by fold — 0.0165, 0.0190, 0.0175, 0.0201, 0.0183 — because market volatility itself varied across those periods. A model’s RMSE has to be compared against the naive baseline’s RMSE within the same fold, not against a single naive RMSE averaged across all five:
| Fold | Naive RMSE | Model RMSE | Skill vs naive |
|---|---|---|---|
| 1 | 0.0165 | 0.0152 | +0.079 |
| 2 | 0.0190 | 0.0201 | −0.058 |
| 3 | 0.0175 | 0.0160 | +0.086 |
| 4 | 0.0201 | 0.0175 | +0.129 |
| 5 | 0.0183 | 0.0188 | −0.027 |
Two of the five folds show a negative skill score even though the mean skill across all five is positive. This is the same fold-by-fold discipline covered for classification metrics in walk-forward validation, fold by fold and ROC AUC, worst-fold performance, and reporting honestly: a single pooled or averaged skill score can present as consistently positive a model that is, in its worst periods, measurably worse than doing nothing.
What a negative skill score should trigger
A negative skill score is not a result to discard quietly and move past. It is diagnostic information: either the model is genuinely unsuited to the series (a real possibility — not every method should be expected to add value on every instrument), or the feature set is capturing noise that happens to fit the training window, a pattern more fully diagnosed against the degradationDegradation The grade given to how much of a strategy's in-sample Sharpe ratio survives out of sample; the statistic recommended as the headline of a backtest. statistic described in walk-forward validation, fold by fold. Either way, a negative skill score is evidence, and it belongs in the report exactly as visibly as a positive one would.