Walk-forward validation, fold by fold

A single chronological split answers one question: did the model work on this particular slice of history? A five-year backtest cut once at the 80% mark produces exactly one out-of-sample score, drawn from exactly one market regime. If that final 20% happened to be an unusually calm or unusually favourable period for the strategy, the single split will not tell anyone. Walk-forward validationWalk-forward validation A sequence of out-of-sample test periods with refitting at each origin, so every reported result is on data the model never trained on. answers a different, harder question: does the model keep working as the estimation window moves forward through time, refitting at each step the way a live deployment actually would?

What a fold actually is

Walk-forward validation partitions the timeline into a sequence of origins. At each origin, the model is refit on everything available up to that point and scored on a subsequent out-of-sample block. Quant Compose runs this as rolling-origin, expanding-window cross-validation by default, with 5 folds. Each fold is purged and embargoed exactly as described in purging and embargoing explained visually — the gap rule applies at every fold boundary, not only once.

Four split modes, ordered by how much dispersion they reveal

The workspace implements four distinct split modes, and the choice between them changes what kind of failure the validation is capable of catching:

1. Single — one chronological split. Fast, but a single number from a single regime.
2. Rolling — walk-forward with a fixed-length training window that slides forward, dropping old data as it adds new data.
3. Expanding — walk-forward with a training window that only grows, never dropping earlier history.
4. CPCVCombinatorial purged cross-validation Many purged, embargoed train/test arrangements built from combinations of time groups, producing a distribution of out-of-sample results instead of one. — combinatorial purged cross-validation, producing many purged, embargoed train/test arrangements from the same data.

Rolling and expanding differ in one respect that matters for interpretation: a rolling window tests whether the model still works when it can only see recent history — closer to how a strategy with a fixed lookback would behave live. An expanding window tests whether more data monotonically helps, and is more forgiving of an early regime the model never has to fully forget. Neither is strictly better; they measure different failure modes, and a model that only passes under an expanding window while failing under a rolling one is a model that depends on total sample size rather than on recent structure — worth knowing before deployment, not after.

What gets reported per fold, and what gets aggregated

For a classification target, each fold reports accuracy, F1, ROC AUCROC AUC The area under the receiver operating characteristic curve; the probability a classifier ranks a random positive case above a random negative one. and balanced accuracy. For a regression target, each fold reports R², RMSE and directional accuracy. Across the fold sequence, the aggregate does not collapse to a single mean — it reports mean, standard deviation, minimum and maximum for every numeric column. That last pair matters more than it sounds: the minimum across folds is the worst foldWorst fold The single lowest-performing fold in a walk-forward or cross-validated evaluation, the number that should headline a validated result, not the average., and it is the number most likely to be quietly dropped from a results summary that only shows the mean.

$$\text{stability} = \frac{\text{mean}(\text{primary metric})}{\text{std}(\text{primary metric})}$$

The primary metric is ROC AUC for classification and R² for regression. A high mean with a high standard deviation produces a low stability score even when the average result looks strong — which is exactly the case where reporting only the mean would be the more flattering, and less honest, choice.

A worked example

Five folds, ROC AUC per fold: 0.61, 0.58, 0.72, 0.55, 0.64.

$$\text{mean} = \frac{0.61+0.58+0.72+0.55+0.64}{5} = 0.62$$

The sample standard deviation across these five values is approximately 0.066, giving:

$$\text{stability} = \frac{0.62}{0.066} \approx 9.4$$

The mean, 0.62, is a genuinely encouraging number for a binary classification task on financial data. The worst fold, 0.55, is barely better than chance. Reporting the mean alone tells a reader the model clears 0.62 consistently; reporting the worst fold tells a reader that at least one out of five out-of-sample periods produced a model that added almost nothing. Both numbers are true. Only one of them is complete.

Fold	ROC AUC
1	0.61
2	0.58
3	0.72
4	0.55
5	0.64
Mean	0.62
Worst (min)	0.55

Degradation: the number that should be the headline

A related but distinct statistic grades how much of a strategy’s in-sample Sharpe ratio survives out of sample:

• Pass: at least 70% retention and fewer than 25% negative folds.
• Warn: at least 40% retention.
• Fail: below 40% retention.

This threshold structure exists because in-sample Sharpe is, by construction, the best score a fitted model could produce on the data it was fit to — it is close to a ceiling, not an estimate. Degradation asks what fraction of that ceiling survives contact with data the model never saw. A strategy reporting an in-sample Sharpe of 2.1 that degrades to an out-of-sample Sharpe of 0.6 has retained roughly 29% — a fail, regardless of how impressive 2.1 looked on its own. This is the number the workspace treats as the headline of a backtest, in preference to the in-sample Sharpe ratio, which on its own answers a question nobody deploying capital actually needs answered.

CPCV: how many splits, how many independent paths

Combinatorial purged cross-validation generalises the single-boundary picture into many boundaries at once. Split the data into $N$ groups, and hold out $k$ of them as the test block for a given split. The number of ways to choose which $k$ groups are held out is $\binom{N}{k}$. The workspace’s default configuration is $(N, k) = (6, 2)$, giving:

$$\binom{6}{2} = 15 \text{ splits}$$

Fifteen distinct train/test arrangements, each purged and embargoed independently. But the 15 splits are not 15 independent equity-curve paths, because the same underlying test group appears across several of the 15 arrangements. The number of genuinely independent paths that can be stitched together is:

$$\varphi = \binom{N-1}{k-1} = \binom{5}{1} = 5$$

Only 5 independent paths. Describing a CPCV result as evidence “across 15 out-of-sample paths” would triple-count the underlying observations and overstate the evidence accordingly. The workspace’s path-stitching logic falls back to reporting split-level dispersion rather than inventing a path count whenever the arrangement, the data, or a mismatched series length would make honest path reconstruction impossible.

Why fold-by-fold beats a single blended score

Collapsing five folds into one number by pooling all out-of-sample predictions before scoring hides exactly the information walk-forward validation exists to surface: whether performance is uniform across time or concentrated in one favourable stretch. A model that scores 0.68 ROC AUC in a pooled evaluation could be produced by five folds clustered near 0.68, or by two folds near 0.9 and three folds near 0.55 — two very different models to deploy, indistinguishable once pooled. Reporting fold-by-fold numbers, alongside the aggregate statistics above, is what makes that distinction visible. The broader discipline of reading a backtest without letting a single flattering summary number stand in for the full distribution is covered in reading a backtest without fooling yourself.