What survives a sceptic
A result the workspace will defend, because it was built to.
Purging, embargoing and a train-only scaler are on by default, not opt-in. Deflated Sharpe and the probability of backtest overfitting refuse to compute without a genuine trial population rather than fabricate one. The Robustness Panel puts raw Sharpe next to Deflated Sharpe on purpose, shows an unavailable statistic as a row that says why, and never sorts by the number that flatters an over-searched configuration.
- Split modes
- 4single, rolling, expanding, CPCV
- Degradation pass threshold
- 70%in-sample-to-out-of-sample Sharpe retention, and under 25% negative folds
- Reproducibility checklist
- 6 itemsdata, parameters, versions, seed, environment, methodology — unweighted
01What you actually do here
Five things, in the order the page presents them.
- 01
Start from defaults that are leakage-safe, not merely available
Chronological split, train-only scaler, no shuffling and purged embargo are all on before a researcher touches a setting. The purge gap is max(embargo, target horizon), floored at the label horizon because the last training rows always share their outcome with the first test rows otherwise.
- 02
Pick a split mode that matches how much dispersion you need to see
Single is one chronological split. Rolling and expanding walk forward with a fixed or growing window. CPCV builds many purged, embargoed arrangements — the default (6, 2) configuration is 15 splits but only 5 independent backtest paths; reporting 15 paths would triple-count every observation.
- 03
Compare against the naive baseline, not against nothing
Forecast diagnostics measure held-out accuracy against a naive last-value-carried-forward baseline, computed the same way regardless of which forecasting method is selected. The skill score is 1 minus the ratio of RMSE to the baseline’s RMSE — a model that cannot beat naive is not adding information, and the workspace reports that honestly rather than hiding a weak result.
- 04
Rank candidates by Deflated Sharpe, never raw Sharpe
Experiment Comparison sorts by Deflated Sharpe because sorting by raw Sharpe is how the most over-searched configuration wins. The Deflated Sharpe Ratio and the probability of backtest overfitting via combinatorially symmetric cross-validation both refuse outright without a real trial population or per-trial return series, rather than estimate one.
- 05
Read the verdict against its own audit trail
Every experiment record carries a SHA-256 hash of its inputs, a write-once verdict, and an environment fingerprint. The trial ledger is genuinely append-only — closing it blocks further appends outright. Deletion of a validated record is gated behind explicit confirmation.
01bThe split, drawn
Purging and embargoing, in ten seconds.
Move the embargo and the label horizon. The purge gap follows the workspace's rule: never smaller than the horizon, because the label on row t already contains prices up to t plus h.
purge gap = max(embargo, horizon) = rows
02How it works
The method, stated for the specialist.
Every statistic in this layer is either computed honestly from real evidence or it refuses to be computed at all. There is no middle setting where a plausible-looking number is produced from an incomplete trial record.
- Multiple-testing and overfitting corrections, source-cited
- White’s Reality Check and Hansen’s Superior Predictive Ability test both take the full set of tried strategies, not just the winner, and bootstrap the null via a stationary block bootstrap — SPA’s measured finite-sample size study is embedded in the code itself. Romano-Wolf stepwise SPA identifies which strategies survive while controlling the familywise error rate. Bonferroni, Holm and Benjamini-Hochberg-Yekutieli haircuts adjust a t-statistic or p-value for how many tests were actually run.
- The Robustness Panel’s three rules
- Raw Sharpe renders directly above Deflated Sharpe so the gap between them is visible, not buried. An unavailable statistic renders as a row stating why, never as a silent absence. Nothing on the panel is ever sorted by raw Sharpe.
- Three verdicts, never a fourth
- Supported, not supported, or insufficient evidence — and the verdict always carries the caveat that it is not investment advice.
- A reproducibility hash, not a tamper-proof chain
- The SHA-256 content hash covers a fixed set of inputs — strategy spec, dataset id and hash, features, universe, period, parameters, split design, seed, engine version — deliberately excluding timestamps and results. It detects "you already ran this exact configuration and got a different answer." It does not link one record to the next the way a blockchain would, and the site never claims that it does.
- Write-once verdicts, gated deletion
- An experiment record that already carries a verdict cannot have it replaced; a different conclusion requires a new experiment. Deleting a record that carries a validation verdict is gated behind explicit confirmation, because removing it removes the evidence behind a claim that may already have been made.
- The reproducibility checklist is six items, unweighted
- Data locked to a content hash, parameters recorded, versions recorded, seed recorded, environment recorded, methodology documented. No composite score — each item is either satisfied or it is not.
03What you get out
Outputs, and what they let you defend.
Validation results are not a page you glance at once. They are the section of the evidence pack a sceptical reader turns to first.
- The Out-of-Sample Validation section of the evidence pack: fold count, mean, standard deviation, worst fold, embargo rows
- A degradation grade — pass, warn or fail — attached to the backtest it describes, not left implicit
- A written verdict (supported, not supported, insufficient evidence) with its caveat
- An environment fingerprint and reproducibility checklist stored with the record
- An append-only trial ledger entry for every configuration actually tried
04What this does not do
Stated plainly, because it matters as much as what it does.
Not tamper-proof
The content hash is a reproducibility-detection tool, not a cryptographic chain. It flags an inconsistent re-run; it does not prevent a determined actor from editing the underlying store.
Statistics refuse rather than estimate
Deflated Sharpe and PBO need a genuine trial population and per-trial returns respectively. Without them the workspace returns a refusal, not an approximate number.
Thresholds are conventional, and recorded
The 70% pass / 40% warn degradation bands are conventional levels, changeable in Settings. Whatever threshold was in force when a result was produced is recorded on that result.
Keep going
Test a result the way a sceptic would.
Beta researchers run the same experiment twice and see whether the ledger agrees with them.