The research loop, end to end

A common way a project goes wrong: someone opens a notebook, pulls today’s list of S&P 500 constituents, trains a classifier on ten years of their price history, and reports a strong backtest. The problem was decided before the model was ever touched. Today’s constituent list excludes every company that was delisted, acquired, or dropped from the index for underperforming — the failures are missing by construction. The model was never given a chance to see them, so it was never given a chance to fail on them either. No amount of careful modelling downstream repairs a universe chosen with hindsight.

This is the argument for treating quantitative research as a loop of dependent stages rather than a single modelling step: universe, then exploration, then data preparation, then portfolio construction, then features, then models, then strategy and backtesting, then evidence. Each stage inherits the decisions of the ones before it, and a mistake made early does not stay contained — it propagates into every number computed afterward.

1. Universe: what you decide to look at

The universe is the set of securities, the index or provider they come from, and the date range under study. Every downstream statistic inherits this choice, survivorship bias included. Consider a simplified illustration: an index currently holds 500 companies. Over the prior 20 years, roughly 300 different companies have cycled through that index as some were removed for poor performance, bankruptcy, or acquisition. A backtest built only from today’s 500 has silently excluded the worst outcomes from the historical sample — the ones most likely to have triggered a stop-loss, a risk limit, or a re-think of the strategy. Reported annual return on such a universe is biased upward for the entire history, not just for a subset of dates. There is no later stage of the research loop — not features, not model choice, not validation — that can add back information the universe excluded before the study began.

2. Explore: look before fitting anything

Before any model is trained, the honest step is to look at what is actually in the data: return distributions, volatilityVolatility The standard deviation of returns over a period, annualised by the square root of the sampling frequency., drawdownDrawdown The decline from a running peak in cumulative value; maximum drawdown is the worst peak-to-trough loss observed in a series. history, and the correlation structure across the universe. This is not a formality. A universe of 40 technology stocks with an average pairwise correlation of 0.65 behaves, for diversification purposes, much closer to a handful of independent bets than to 40. A researcher who skips exploration and goes straight to feature engineering can build an elaborate model on top of a data structure they never actually characterised — for instance, discovering only after the fact that half the sample sits inside a single volatility regime, so a “robust” result is really a description of one market condition.

What a model will later be asked to explain is worth understanding on its own terms first: what is the shape of returns (fat tails, skew), how much of the variance is common across the universe versus specific to each name, and where the extreme observations sit in time. This groundwork gives later results a frame of reference — a Sharpe ratio of 1.2 means something different in a universe with 12% annualised volatility than in one with 35%.

3. Data: the split decides what the model is allowed to see

Preparing data means aligning the trading calendar across securities, deciding a policy for missing or duplicate rows, and — the step with the most consequence for everything after it — setting the chronological split between the data a model trains on and the data it is tested against. A split that shuffles rows at random, rather than keeping them in time order, lets rows from the future sit in the training set and rows from the past sit in the test set. Every metric computed from that arrangement is inflated, because the model has effectively been shown the answer before the exam.

Two further mechanics protect the boundary between train and test even when the split is chronological. A purgePurge Removing training rows whose label window overlaps a test block, so a model never trains on information the test period's outcome depends on. gap removes training rows whose outcome window overlaps the test period — if a label is built from prices five days ahead, then the training rows within five days of the test boundary already “know” something about it. An embargoEmbargo Removing the rows immediately after a test block, where serial correlation would otherwise let information leak backward into training. removes additional rows immediately after the test period, where serial correlation in returns would otherwise let information leak backward. A common implementation sets the purge gap to the larger of the embargo length and the label horizon — a five-day-ahead label needs at least a five-row gap regardless of how the embargo is configured, because the label itself is built from data the test period has not reached yet.

Worked example: with a five-day forecast horizon and an embargo set to two days, the effective gap is $\max(2, 5) = 5$ rows, not two. A researcher who only configured the embargo and assumed that was the whole safeguard would have three rows of unprotected overlap.

4. Portfolio: weights under constraints, then stress

Once a universe and a validated split exist, portfolio construction decides how capital is allocated across the securities under study — subject to constraints such as position limits or turnover budgets — and then subjects that allocation to hypothetical shocks or a replay of a real historical episode. A strategy that clears its backtest still has to survive its portfolio: a signal that looks strong security-by-security can still produce a portfolio with concentrated risk in one sector, or one whose diversification benefit evaporates exactly when a historical stress episode hits every holding at once. Testing an allocation against, for instance, the 2008 financial crisis window on the security’s own historical prices for those exact dates shows whether the diversification the portfolio construction step assumed actually held up when it mattered, rather than only in calmer periods.

5. Features: hypotheses, not facts

A feature — a momentum measure, a volatility ratio, a valuation multiple — is a hypothesis about what drives returns, encoded as a number a model can consume. Building the feature does not confirm the hypothesis; it only makes the hypothesis testable. A twelve-month price-momentum feature encodes the hypothesis “recent relative performance persists”; whether that hypothesis holds in a given universe and period is exactly what the modelling stage exists to check. Treating a feature as though its existence already proves its usefulness — building forty features and assuming several must matter — is how noise gets mistaken for structure before a model has even been fit.

6. Models: candidates, not answers

A model is a candidate explanation, and it earns that status only by being compared against alternatives and against doing nothing at all. The comparison against doing nothing is the naive baselineNaive baseline The simplest rule that requires no model at all — predict no change, predict the trend continues, predict the historical average — that any predictive claim must beat. — for a forecasting task, a rule as simple as carrying the last observed value forward. Skill is measured as the reduction in error relative to that baseline:

$\text{skill} = 1 - \frac{\text{RMSE}_{\text{model}}}{\text{RMSE}_{\text{baseline}}}$

A skill score of 0.08 means the model reduced forecast error by 8% relative to doing nothing — modest, but positive. A skill score at or below zero means the model added no information a naive rule did not already provide, regardless of how sophisticated the model architecture is.

For classification tasks the same discipline applies through cross-validated ROC AUCROC AUC The area under the receiver operating characteristic curve; the probability a classifier ranks a random positive case above a random negative one. rather than a single in-sample accuracy figure, reported with its spread across folds so a mean score does not hide a model that only works in some periods.

7. Strategy and backtest: two different questions

Turning a model’s output into a strategy means specifying a threshold, a direction, a position size, and risk controls such as stops and drawdown halts — then testing the result net of transaction costs and slippage against a stated benchmarkBenchmark A stated comparison point — an index, an equal-weight allocation, a risk-free rate — that a result must be measured against before it can be called good.. Classification accuracy and financial value are different questions, and a model can score well on one while failing the other. A classifier with 58% directional accuracy can still lose money once realistic costs are applied, if the correct calls are concentrated in small moves and the wrong calls in large ones. A five-basis-point commission charged on every executed change in exposure is easy to ignore in a spreadsheet and decisive over a year of frequent rebalancing — a strategy that trades weekly at 5 bps commission and 2 bps slippage pays roughly $52 \times 7\text{bps} = 3.64\%$ a year in frictions before it earns anything, a cost that a costless backtest never shows.

8. Evidence: a number someone else can check

The final stage assembles universe, statistics, portfolio results, features, model performance, and the backtest into a document — a claim, methodology, results, and stated limitations — that another person can trace back to what actually produced it. A number that cannot be traced to its data, parameters, and code version is a number that cannot be defended later, whether that later is a professor, a risk committee, or the researcher’s own future self revisiting the project in six months.

Two of the eight stages carry the most leverage over everything that follows: the universe — what quantitative research actually is covers what counts as a defensible one — and the data split, where signal versus noise explains why most inflated results are actually manufactured. Getting the earliest stages right costs less time than discovering, after building a model, that the foundation it stands on was never sound.