adduce

A paper reports 81.4% accuracy. Six months later a reviewer, or the author, tries to find the run that produced it. The config that trained the checkpoint has drifted, the seed was never set on two of three RNG sources, the dependency list has no lockfile, and the model it pulled from a hub was never pinned to a revision. Nothing here is fraud. It is ordinary entropy, and it is invisible until someone tries to re-run the work.

Reproducibility checklists ask authors to attest to all of this from memory, at submission time, in a form nobody can verify. adduce inverts that: it reads the repository and reports what is actually observable in it.

The question it is built around: for every number in the paper, can I point to the artifact that produced it, and will that artifact still produce it elsewhere?

adduce check . runs 78 rules across 17 categories over code, configs, data, seeds, environment, remote artifacts, and generated results, then reports a score, a reviewer-time estimate, and a ranked list of fixes. It is offline and static by default.

pipx install adduce
adduce check .

Every finding carries a status (pass / partial / fail / not-applicable / unknown), a confidence, a file:line location where one exists, and a concrete remediation. partial exists because most real repositories half-satisfy most checks, and collapsing that into pass/fail throws away the only information an author can act on.

Beyond the audit it drafts the submission material that the audit already has evidence for: repository-observable NeurIPS/ACL checklist items, an ACM Artifact Appendix, archival metadata (RO-Crate, Croissant, CodeMeta, Zenodo), and a claim-by-claim evidence trail.

adduce check on [nanoGPT](https://github.com/karpathy/nanoGPT) at commit 3adf61e, scoring 54/100. Six of the fourteen categories that applied:

Reviewer time to first result: 23-83 minutes, rated Risky — no one-command reproduction path, environment assembled by hand, dependency resolution may not converge.

The findings are anchored, not abstract: the TF32 finding points at train.py:107, the unpinned hub call at model.py:238. Where a manifest declares claims, the report adds a per-claim trail that walks from the sentence in the paper to the logged metric that supports it, and flags a rounding gap between a stated 81.4 and a logged 81.37 as PARTIAL rather than silently accepting or rejecting it.

Static and offline by default. The whole tool runs without touching the network or executing the repository. Public-metadata lookups require an explicit --online or pin-remotes flag and go through a bounded public-HTTPS resolver; dynamic reproduction is a separate opt-in layer. Cost: static resolution has a ceiling, and a dynamically-constructed config path will defeat it. The alternative — executing untrusted research code by default to find out — is worse.

Pre-existing cache entries never count as network evidence. A cached response proves the artifact resolved once, not that it resolves now. Counting it would quietly convert a stale cache into a passing check.

Applicability gating over a flat rubric. Each rule is gated on whether it applies, so a repository with no notebooks drops the Notebooks category out of scoring rather than scoring zero on it. Without this, the score mostly measures project shape rather than project rigour.

Rules as a plugin API, not a fixed list. Every field has conventions the core cannot encode. A plugin API and a GitHub Action mean a lab can add its own rules and gate PRs on them.

The hardest design constraint was resisting the tool's own conclusions. A number between 0 and 100 next to the word "reproducibility" reads as a verdict no static analyser can support.

So adduce never says a repository is reproducible. It reports detected signals. It does not assess execution-based badges (Results Reproduced / Replicated) at all — only badge eligibility signals. Scores and tiers are labelled experimental prioritisation aids, and the validation corpus protocol defines a release-quality gate that is explicitly still pending: no effectiveness or calibration claim is made until its human-review requirements are complete. Automatically inferred claims stay marked [inferred draft] until an author confirms them, and generated submission material is always a draft.

Shipping docs/honest-limits.md as a first-class page, and pointing at it from the README, was a deliberate choice. The failure mode for a tool like this is being trusted more than it has earned.

Published on PyPI at 0.1.2, MIT licensed, CI green, beta. Validated across 15 repositories plus a synthetic corpus with positive and negative controls — the synthetic_rounding_match fixture above exists so that the rounding-tolerance path has a test that fails loudly when it regresses.

• Calibration against human review. The scores are currently ordinal at best. The corpus protocol needs enough manually-reviewed real repositories to say what a 54 means relative to a reviewer's judgment, which is the gate before any effectiveness claim.
• Raise the static-resolution ceiling. Config values assembled at runtime, dynamic imports, and indirection through helper factories are where the analyser gives up and returns unknown. Limited dataflow analysis would convert some of those into real findings.
• Historical remote recovery. pin-remotes is a forward guarantee: it pins what resolves today. It cannot recover what a hub reference pointed at two years ago, which is exactly the case where an author most needs the answer.
• Lower the false-positive rate on the fix ranking. The top-fixes list is ordered by score gain, which is not the same as ordered by reviewer impact. Weighting by the reviewer-time model rather than raw points would make the first suggestion the right one more often.