Why random train/test splits break on time series
Random k-fold cross-validation assumes independent rows. Financial data never is. Here is the arithmetic behind why that assumption fails.
A student trains a classifier to predict next-week direction on five years of daily equity data. Standard scikit-learn workflow: shuffle the rows, split 80/20, fit, score. The model reports 61% accuracy — better than a coin flip, publishable-looking. Then someone reruns the same model with a chronological split instead of a shuffled one, and accuracy drops to 51%. Nothing about the model changed. The number that changed was the only number that mattered.
This is not a bug in the model. It is a property of how the split was drawn, and it is the single most common way a promising-looking backtest turns out to be measuring nothing.
The assumption k-fold cross-validation makes
Ordinary k-fold cross-validation, and the random 80/20 split that precedes it in almost every introductory tutorial, rests on one assumption: that each row is drawn independently from the same distribution. Shuffle the rows, and independence means shuffling changes nothing about what the model can learn from them. That assumption is true for a dataset of, say, handwritten digits. It is false for a dataset of daily stock returns.
Two things break it.
Serial correlation. Yesterday’s volatility predicts today’s volatility. Yesterday’s price is highly informative about today’s price. Rows drawn from adjacent days are not independent draws from an underlying distribution — they are consecutive samples from a single evolving process. A model that sees day t in training and is tested on day t+1 is not being asked to generalise to a new situation. It is being asked to interpolate between two points it has effectively already seen.
Overlapping label horizonsLabel horizon The number of future periods a label looks ahead to determine its outcome; it sets the minimum purge gap a validation split must respect.. Most predictive targets in finance are built from a forward window — “will the price be higher in 5 days,” “what is the 10-day forward return.” A label built at row t depends on prices up to t + h. If the label horizon is 5 days and the split puts row 500 in training and row 502 in the test set, the training label for row 500 was computed using price information through day 505 — three days after the test point at row 502 begins. The training set has, in effect, already seen part of the future the model is being tested against.
A worked example
Take a dataset of 1,000 daily observations, target horizon (a 5-day-forward return sign), and a naive random 80/20 split with no purging. Because the split is drawn independently and uniformly at random, any given test row has an 80% chance that each of its horizon-adjacent neighbours — the rows immediately before it and the rows immediately after it — lands in the training set rather than the test set.
For a single test row, the number of neighbouring rows within the label horizon is . The probability that none of those 10 neighbours falls in the training set is:
In other words, for essentially every test row, at least one of its label-overlapping neighbours sits in the training set. The model was not evaluated on unseen data — it was evaluated on data whose labels share raw information with rows it trained on. The 61% accuracy in the opening example was not skill. It was leakageLeakage Any pathway by which information from outside the training window reaches a model during fitting, inflating every metric that follows., and it was leakage baked into the sampling procedure itself, not a coding mistake anyone could spot by reading the model code.
This is why Quant Compose’s data pipeline treats a chronological split as the default rather than an option: chronological_split: true, no_shuffling: true. Turning either off does not just weaken the evaluation — it deliberately converts a leakage-aware walk-forward setup into ordinary random-sample cross-validation, rows permuted with a fixed seed. The workspace allows this because a researcher studying the effect of shuffling might want to reproduce it deliberately; it warns loudly because reproducing it by accident is the default way a backtest lies.
What chronological ordering fixes, and what it does not
Switching to a chronological split — train on the first 80% of dates, test on the last 20% — removes the shuffling problem outright. Rows in the training window can no longer be drawn from the future relative to the test window as a matter of construction.
It does not, by itself, remove the label-horizon overlap at the boundary between train and test. The last few training rows before the split date still have labels built from prices that extend into the test period, for exactly the reason described above. A chronological split is necessary but not sufficient. The remaining gap between “no shuffling” and “no leakage” is closed by purgingPurge Removing training rows whose label window overlaps a test block, so a model never trains on information the test period's outcome depends on. and embargoingEmbargo Removing the rows immediately after a test block, where serial correlation would otherwise let information leak backward into training. the rows immediately around the split boundary — the subject of purging and embargoing explained visually.
Scaling makes the same mistake in a different place
The same independence assumption quietly reappears in feature scaling. A StandardScaler fit on the full dataset before splitting computes its mean and standard deviation using test-period values — the scaler has seen the future, even if the model technically has not. Quant Compose’s pipeline fits the scaler on the training split only (scaler_train_only: true) and refits it per fold inside walk-forward validation, so the transform applied to a test row was never informed by that row or any row after it. This matters more than it looks: a feature like rolling volatility, standardised against a window that includes a later volatility spike, will systematically understate how extreme early-period observations were — an effect that shows up as spuriously stable feature distributions and, eventually, as a model that looks calibrated in-sample and is not.
Table: what changes between a random split and a leakage-safe one
| Property | Random split | Chronological, purged split |
|---|---|---|
| Row order | Shuffled | Preserved |
| Scaler fit | Often on full data | Train only, refit per fold |
| Label-horizon overlap at boundary | Untreated | Removed by purge gap |
| Serial-correlation leak after boundary | Untreated | Removed by embargo |
| Reported accuracy | Optimistic, uninterpretable | Traceable to a specific out-of-sample window |
Why this is worth getting right before touching a model
None of this is about which model family to choose. A random forest, a logistic regression and a gradient-boosted tree are all equally vulnerable to a bad split, and all equally capable of reporting an impressive-looking score against one. The split design determines what the reported number is even measuring. Fixing the split after finding a good score is not a validity check — a researcher who tunes a model against a leaking split first and only then re-validates chronologically has already spent their best guesses against the leak, and the chronological re-run will look worse for reasons that have nothing to do with an honest sanity check. The split has to be right from the first experiment, not retrofitted once a number looks worth defending.
The broader pattern — a result that survives one way of looking at the data and collapses under another — is covered in signal versus noise: what a result has to survive. A random split is one of the easiest ways to manufacture a result that looks like signal and is, in fact, an artefact of the sampling procedure.