A paper's code can sit in a repository for years and never be run again. When someone finally tries, the result is often a stack of errors rather than a number. The fix is unglamorous: rebuild the archived environment, execute the analysis, and compare outputs against the published tables before treating the finding as settled.

The Silent Decay of Archived Code

Computational results are unusual among scientific claims because they depend on a machine state that no longer exists. A simulation tuned to a particular compiler, a statistical model relying on a library default, a plotting script that once produced figure three — each is a snapshot of an environment that has since moved on. The paper stays fixed. The runtime does not.

Deposit requirements have multiplied without keeping pace. Journals and funders increasingly ask for code alongside data, and repositories such as Zenodo and Software Heritage will hold it indefinitely. Storage is solved. Execution is not. A deposit that has never been rebuilt is a claim about a file's existence, not about a result's reproducibility.

Dependencies break quietly. A numerical library changes a default tolerance, a language drops a deprecated function, an operating system ships a different math library. None of these announce themselves. The code still looks correct in the archive, and the published number still looks correct in the table, and the two may no longer have any relationship.

The decay is not evenly distributed. Projects with few dependencies and a stable numerical core age well; sprawling pipelines that chain half a dozen packages, call out to a database, and read a configuration file from a home directory tend to fail first. Age since publication is a rough predictor, but the number of moving parts is usually a better one. A three-year-old project with forty transitive dependencies can be harder to resurrect than a decade-old script that needs nothing beyond a standard library and a compiler.

What Reproducible Builds Reveal

Reproducible builds, sometimes called deterministic compilation, are the practice of making a build produce byte-identical output from identical source. The idea comes from software supply-chain work, where it lets independent parties verify that a distributed binary matches its source. In a research setting it functions as an instrument: if two rebuilds disagree, something in the environment is doing work nobody documented.

That disagreement is the diagnostic. Bit-for-bit comparison turns a vague worry about drift into a specific diff, pointing at a library version, a compiler flag, or a timestamp embedded in an output file. The same logic applies to analysis pipelines that produce tables rather than binaries. Hash the output, rebuild, and compare.

Timestamps deserve a mention of their own, because they are the most common false alarm. Many build systems embed the current date, a build path, or a random seed into their output, so two runs of identical source produce different bytes for reasons that have nothing to do with the science. Reproducible-build practice handles this by normalising those fields, and the same normalisation is worth doing before you conclude that a numerical result has drifted. A diff that turns out to be a build path is a distraction; a diff that turns out to be a changed summation order is the real thing.

The instrumentation angle matters more than the security one here. Provenance records answer who built what and from which commit. They do not answer whether the archived project still executes. Those are separate questions, and the second is the one a reader of a five-year-old paper actually needs answered.

Rebuilding Old Analyses From Scratch

Containerization has made the practical side easier. A container image can pin an operating system, a language runtime, and every library at the versions used at publication, and it can be stored next to the paper. The catch is that the image itself must be built and tested, not merely uploaded. An untested container is a more elaborate form of the same unverified deposit.

Audits that re-run published analyses tend to find high failure rates, though the figures vary widely by field and by how strictly the rerun is defined. Some failures are trivial, a missing file path or a renamed column. Others are substantive: the pipeline runs, produces output, and the output no longer matches the reported table. The second category is the one that should worry a reader.

The definition of success matters as much as the failure count. A rerun that reproduces a table to three decimal places is a different claim from one that reproduces the qualitative conclusion but shifts the effect size. Audits that report a single headline percentage usually collapse those cases together, which makes the number hard to interpret and easy to quote out of context. Read the methods of any such audit before you read its headline: what counted as a reproduction, how many attempts were allowed, and whether the original authors were contacted all change the denominator.

When a rerun does succeed, the effect often shrinks. This pattern shows up across computational reanalyses, and it is not always a sign of error in the original. Analytical choices that were reasonable at the time can shift a coefficient, and a rerun that fixes those choices may land somewhere different. The point of the exercise is to know which situation you are in.

The Cost of Assuming It Still Works

Floating-point pipelines are especially prone to silent drift. Changing the order of a summation, swapping a linear algebra backend, or updating a compiler's optimization settings can alter the last digits of a result. Usually that is harmless. Occasionally a threshold comparison flips, a fit fails to converge, and a reported classification changes. Nothing in the code review would catch it.

Library defaults are a second source of quiet change. Statistical packages revise default priors, optimizer settings, and handling of missing values between major versions. A script that never specifies these explicitly inherits whatever the installed version decides. The author knew the defaults in force at publication. A later reader does not.

Hardware belongs on the same list. A pipeline that ran on one vendor's math library may produce slightly different digits on another, and routines that were tuned for a particular processor generation can take a different path through the same code. None of this is visible in the source. It surfaces only when the same inputs are pushed through a different machine and the outputs are compared, which is exactly the comparison an archive that has never been rebuilt cannot supply.

Peer review rarely includes execution. Reviewers read methods sections and check figures against claims; few rebuild the analysis, and fewer still do so in a clean environment. Trust therefore accumulates faster than verification, and a result can be cited dozens of times before anyone discovers that its code no longer runs. This site has documented a related pattern in replication audits that track researchers, where the audit design shapes what gets found.

A Protocol for Runtime Verification

Verification works best as a routine rather than a rescue operation. The steps are not technically demanding, but each one has to be done deliberately, and the record of what happened is as valuable as the pass or fail.

1. Deposit the source, the full environment specification, and the build recipe together, so the runtime can be reconstructed without guesswork.
2. Execute the deposit at the moment of archiving, and record whether it reproduces the published tables and figures.
3. Re-run the check at fixed intervals, and log the failure mode when one appears rather than only the outcome.
4. Treat the runtime check as part of the result, cited alongside the code and the data.

Recording failure modes is the part most often skipped. A binary pass or fail tells you little; a note that the pipeline breaks at a specific library call, or that it runs but shifts a coefficient by a known amount, tells the next reader exactly where to look. That record is cheap to produce at deposit time and expensive to reconstruct later.

The trade-off is real. Maintaining a runnable environment across a decade of library churn is ongoing work, and it competes with new research for the same hours. A lab that pins aggressively may find itself unable to adopt performance improvements. The argument for doing it anyway is that a result nobody can execute is a result nobody can check.

There is a middle path worth naming, because the all-or-nothing framing discourages people from starting. Pin the environment for the analysis that produces the headline result, and let the exploratory scripts drift. Keep a single tested image rather than a fleet of them. Re-check on a schedule that matches how often the result is likely to be cited rather than on a fixed annual cycle. The goal is not a perfectly preserved museum of every script a lab ever wrote; it is a small number of tested entry points that a stranger can execute and compare.

What to Do Before Trusting a Result

Clone the archive into a clean environment and rebuild it there, rather than adding it to a working setup that already has the right libraries installed. Compare the outputs against the reported tables and figures directly, not against a summary of them. Check the dependency pins and any system-level libraries the pipeline calls, since these are the usual culprits when a rebuild behaves differently. Document every deviation from the published run, including the ones you decide are harmless. Ask the authors for the exact runtime they used, and treat a vague answer as information about how much the result has been verified.

Keep a record of what you tried, even when the rebuild succeeds on the first attempt. The next person to open the archive will benefit from knowing which environment worked, which packages had to be held back, and which warnings were present but benign. That log is a few lines long and saves hours. It also turns a private troubleshooting session into a public asset, which is the difference between an archive that gets used and one that gets opened once and abandoned.

A related piece on this site makes a comparable point about simulations below grid resolution: the numerical setup determines what you can see, and the setup is easy to lose. Runtime verification is the same discipline applied to the whole pipeline rather than one parameter. It does not settle whether a finding is true. It settles whether the finding still computes, which is where any serious check has to start.