Reproducible research: snapshots, hashes and audit trails

A researcher reruns a backtest six months after first presenting it, because a colleague wants to check the Sharpe ratio before citing it in a memo. The number that comes back is different. Not wildly different — 1.31 instead of 1.24 — but different enough that the memo now needs a footnote nobody wants to write. Was the dataset refreshed since the original run? Did a dependency update change how volatility gets annualised? Did someone quietly widen the universe? Without a record of exactly what produced the first number, there is no way to answer any of these questions, and the memo’s credibility rests entirely on memory.

This is the problem reproducibility solves. Not “the code still runs” — that is a much weaker claim — but “the same configuration, run again, produces the same answer, and if it doesn’t, the workspace notices.”

What a content hash actually proves

Every experiment record in the workspace carries a SHA-256 content hash computed from a fixed set of inputs: the strategy version and specification, the dataset id and its own content hash, the feature set, the universe, the period, the parameters, the split design, the stated assumptions, the benchmark, the random seed, and the engine version. Timestamps, record ids and results are deliberately excluded from what gets hashed.

That exclusion is the whole point. A hash that included the result would tell you nothing — of course a record’s hash would match itself. A hash built only from inputs lets the workspace ask a genuinely useful question: has this exact configuration been run before, and if so, did it produce the same answer?

This distinction matters because the two claims get conflated constantly, usually by people trying to sound more rigorous than the mechanism supports. “Blockchain-verified” and “tamper-proof” are strong claims about the integrity of a historical sequence. “Same inputs produce the same output, and a mismatch gets flagged” is a narrower, checkable claim about determinism. The workspace makes the second claim, not the first, and the difference is worth stating plainly to anyone who asks.

Consider what the hash actually catches. A researcher reruns an experiment and the recorded inputs match a prior record byte for byte — same dataset id and content hash, same parameters, same split design, same seed — but the result differs. That is not supposed to happen, and it points to something real: a library version drift, a non-deterministic operation that wasn’t seeded, or a silent change in how a dependency computes a step in the pipeline. The mismatch is the signal. A researcher who never checks this can publish two different Sharpe ratios for “the same” backtest and never learn why they disagree.

The verdict, written once

A validation verdict — the pass/warn/fail judgment attached to an out-of-sample result — is write-once once it exists on an experiment record. It cannot be edited to a different verdict after the fact. If the underlying analysis changes, or someone disagrees with the original call, the answer is a new experiment record with its own verdict, not an edit to the old one.

This sounds inconvenient until you consider the alternative. A verdict that can be silently changed is a verdict nobody can trust six months later — the number in a report might reflect the original judgment, a later revision, or something in between, and there is no way to tell which from outside. Write-once verdicts trade a small amount of convenience for something a reviewer actually needs: the guarantee that what they are reading is what was originally decided, not a retouched version of it.

The ledger that never forgets a trial

Every time a researcher tries a configuration — a parameter set, a feature combination, a universe cut — that attempt is recorded in a trial ledger. The ledger is genuinely append-only: there is no update operation and no delete operation for an individual trial once it is recorded. A ledger can be closed, after which further appends are refused outright, but a closed ledger cannot be reopened and edited.

The reason this matters is not abstract. If a researcher tries forty parameter combinations and only reports the best one, the reported Sharpe ratio is not a measure of skill — it is a measure of how many chances the search was given to get lucky. A statistic like the Deflated Sharpe Ratio exists specifically to correct for this: it takes the number of configurations actually tried and discounts the headline result accordingly. But that correction only works if the trial count is real. A ledger a researcher can quietly edit down to “the trials that mattered” defeats the entire purpose of counting them. Append-only is not a compliance flourish here — it is the property that makes the correction meaningful at all.

The audit trail that ties a trial to the run it came from is what makes the ledger checkable at all — see /quantlab/foundations/failure-modes-in-retail-quant for what happens when trial counts go unrecorded in the first place. This article assumes the counting is already happening and focuses on why the ledger has to be tamper-resistant to be worth anything.

Deleting an experiment record that already carries a validation verdict is gated behind an explicit confirmation step, precisely because removing it removes the evidence behind a claim that may already have been made elsewhere — in a report, a memo, a thesis chapter. Bulk deletion only ever targets records that were never validated in the first place, so a researcher tidying up scratch experiments cannot accidentally erase evidence a real conclusion depends on.

Snapshots freeze a view, not the world

A related but distinct mechanism is the saved result, sometimes called a snapshot: a captured copy of one panel or page as it appeared at a moment in time, stored with the parameters in force at capture, the underlying figures, and where it came from — project, route, folder, title, timestamp.

A snapshot is deliberately a dead end in one direction: it does not capture live application state. There is no stored record of which pipeline steps were selected, which filters were active, or what the researcher was about to do next. Reopening a snapshot shows the frozen markup or image exactly as it was captured. It does not repopulate the page’s controls, and it does not rerun anything. This is a narrower guarantee than it might sound — a snapshot proves what a result looked like, not that it can be regenerated with one click.

The six-item reproducibility checklist

The workspace tracks reproducibility as an unweighted checklist rather than a single composite score, because a composite score invites exactly the kind of false precision this whole topic exists to avoid. The six items:

Item	What it confirms
Data locked	Dataset id and its content hash are recorded, so the input data is identifiable, not just described
Parameters recorded	Every parameter used in the run, including resolved defaults, not only what the user typed
Versions recorded	The code and engine version that produced the result
Seed recorded	The random seed, so any stochastic step can be replayed
Environment recorded	The runtime context the analysis executed in
Methodology documented	Split design, embargo setting and stated assumptions

There is no weighting because there is no meaningful way to trade one item off against another — a result missing its seed is not “80% reproducible” if four of the other five items are present. It is simply missing something a future reader will need. The checklist exists to make the gap visible, not to produce a score that looks better than the underlying evidence.

This connects directly to how a finished report should be structured. A report that states a result without any of these six items attached is asking a reader to take the number on faith, which is the opposite of what a checkable evidence pack is for — see /quantlab/practice/writing-a-checkable-report for how the fixed evidence pack turns this checklist into something a reader actually sees, section by section.

Putting it together

None of these mechanisms individually make a result true. A content hash cannot tell you the strategy is profitable. A ledger cannot tell you the parameter search was well designed. A checklist cannot tell you the split respected chronology. What they do collectively is remove the excuses for a result being unaccountable: the inputs that produced it are identifiable, the trials that led to it are counted and cannot be quietly pruned, the verdict attached to it cannot be revised after the fact, and a reader six months later can check every one of those things without asking the original researcher to remember.

That is a modest claim, deliberately. A workspace that promised more — that called its hash a tamper-proof chain, or its checklist a reproducibility score — would be making a claim it cannot back up. Naming the mechanism precisely, and naming what it does not do, is the more defensible position, and it is the one a sceptical reviewer will actually trust.

Key takeaways

The input content hash detects a re-run that produces a different answer from identical inputs; it is not a tamper-proof chain linking records together.
Validation verdicts are write-once: a different conclusion requires a new experiment record, not an edited old one.
The trial ledger is append-only with no update or delete for individual trials, which is what makes trial-count corrections like the Deflated Sharpe Ratio meaningful rather than gameable.
A saved snapshot freezes what a panel showed at a moment in time; it does not capture live application state and cannot be edited, only annotated.
Reproducibility is tracked as an unweighted six-item checklist — data, parameters, versions, seed, environment, methodology — because a composite score would hide exactly the gaps the checklist exists to surface.