Forecasting versus prediction: what you can and can't claim

A student builds a Random Forest Lags model, forecasts tomorrow’s closing price, and reports an R² of 0.97. It sounds like a strong result until a second student points out that a model which simply repeats today’s closing price as tomorrow’s forecast would score nearly the same R², because daily closing prices barely move relative to their own level. The 0.97 was never evidence of skill — it was evidence that prices are highly autocorrelated at one-day lags, which every forecasting model, including a trivial one, exploits automatically. This is the single most common way a forecasting result misleads, and the fix is a fixed, honest baseline every forecast has to beat.

Forecasting is not the same claim as prediction

“Prediction” in the ML Models page context usually means classifying or estimating an outcome from a rich, engineered feature set — momentum, volatility, cross-sectional rank, technical indicators — evaluated on a chronological, leakage-aware split. “Forecasting,” in the Model Forecasting page’s specific sense, means something narrower and more disciplined: producing the next value of a single time series from its own recent history, using real lags of that series and nothing else. A forecast of tomorrow’s price built from today’s, yesterday’s and the day before’s price is a genuine one-step-ahead forecast; a forecast built from a feature engineered using information that would not have existed at the time being predicted is not a forecast at all, it is leakage wearing a forecast’s name.

Four forecasting methods are available on this dedicated code path: the Naive method, which simply carries the last observed value forward; Linear Lag Model, which fits a linear combination of a fixed number of recent lags; Random Forest Lags and XGBoost Lags, which fit the same style of lag features through a tree ensemble instead of a linear combination. All four are evaluated identically, against the same one-step-ahead target, which is what makes comparing them meaningful.

The naive baseline is not a strawman

The naive baseline — “tomorrow equals today” — is not included as an easy target to beat for show. It is included because it is often genuinely difficult to beat on financial series, and that difficulty is itself the finding. Daily closing prices are close to a random walk: today’s price already incorporates almost everything currently known about where the price should be, so the single best guess for tomorrow, absent new information, really is close to today’s value. A forecasting model earns its complexity only by adding something the naive baseline does not already have, and the size of that addition is what the naive comparison is built to measure.

The skill score

The comparison is formalised as a skill score:

$\text{skill}_{\text{vs. naive}} = 1 - \frac{\text{RMSE}_{\text{model}}}{\text{RMSE}_{\text{naive}}}$

A skill score of 0 means the model’s forecast error is identical to the naive baseline’s — no information added. A positive skill score means the model’s RMSE is smaller than the naive baseline’s; a skill score of 0.20 means the model’s forecast errors are, on average, 20% smaller in magnitude than simply carrying today’s value forward. A negative skill score means the model is doing worse than doing nothing at all, which is a real and informative outcome — not a bug, and not grounds for tuning the model until the sign flips.

A worked example makes the arithmetic concrete. Suppose ten one-step-ahead forecasts on an out-of-sample window produce the following errors:

Day	Naive absolute error	Model absolute error
1	0.42	0.38
2	0.15	0.20
3	0.88	0.51
4	0.31	0.29
5	0.09	0.11
6	0.67	0.44
7	0.22	0.19
8	0.54	0.48
9	0.18	0.22
10	0.39	0.33

The naive RMSE is $\sqrt{\frac{1}{10}\sum e_i^2} = \sqrt{0.2051} \approx 0.4529$ . The model’s RMSE is $\sqrt{\frac{1}{10}\sum e_i^2} = \sqrt{0.1158} \approx 0.3403$ . The skill score is:

$1 - \frac{0.3403}{0.4529} \approx 0.249$

The model reduced forecast error by roughly 25% relative to carrying today’s value forward — a genuinely useful, checkable claim, in contrast to reporting the model’s raw RMSE of 0.3403 alone, a number with no fixed scale that tells a reader nothing about whether 0.34 is impressive or mediocre for this particular series.

Directional accuracy: a different, complementary question

Skill score answers “how much smaller is the average error.” A separate and often more decision-relevant question is directional accuracy: of all one-step-ahead forecasts, what fraction correctly called whether the series would rise or fall, independent of the forecast’s magnitude error. A model can have a modest skill score and still have a directional accuracy meaningfully above 50%, which matters more than magnitude accuracy for a strategy that only consumes the sign of the forecast. The reverse is also possible — a model can shave meaningful RMSE off the naive baseline through better handling of large, obviously-directional moves while adding no real information about the more frequent small day-to-day moves whose direction actually determines a trading signal’s usefulness. Reporting both figures side by side, rather than either alone, is what lets a reader judge which claim the model is actually supporting.

What separates a defensible forecasting claim from an indefensible one

Three checks distinguish a forecasting result that survives scrutiny from one that does not. First, the lags used must be genuine — a lag of the series’ own past values, not a feature that incorporates information not yet available at the time being forecast, which is the same look-ahead discipline covered from the feature-engineering side in look-ahead bias and data leakage. Second, the comparison must be against the naive baseline computed on the identical out-of-sample window, not against an arbitrary or self-selected reference point. Third, the reported figure must be the skill score or the directional accuracy, not a raw error metric or an R² on price levels, because only the relative figures say anything about whether the model added information beyond what a trivial rule already captures.

None of this is unique to forecasting — it is the same discipline explored for classification and regression models generally in beating the naive baseline, applied here to the specific, narrower case of one-step-ahead time-series forecasting from a series’ own lags.

Key takeaways

A one-step-ahead forecast is built from genuine lags of a series’ own history; a feature built from information not yet available at forecast time is leakage, not forecasting.
The naive baseline — carrying the last observed value forward — is often genuinely difficult to beat on financial series, which is exactly why it is the correct comparison rather than a strawman.
The skill score, $1 - \text{RMSE}_{\text{model}} / \text{RMSE}_{\text{naive}}$ , expresses how much smaller a model’s forecast error is relative to doing nothing; zero or negative means no information was added.
A high R² on raw price levels is close to meaningless, because the naive baseline typically scores an equally high R² on the same autocorrelated series.
Directional accuracy and skill score answer different questions and should be reported together, since a model can excel at one without excelling at the other.