Nine ways a paper tells you it won’t reproduce – before anyone tries
The remedy to the reproducibility problem lies in the hands of those who steward the record, writes Mayank Vadaliya
Machine learning ought to be the easy case for reproducibility. Its objects of study are code, data, and configuration – there are no reagents to age, no organisms to vary, no delicate bench technique to master. And yet the field has a reproducibility problem that is by now thoroughly documented. In the most widely cited cross-disciplinary survey, more than 70% of researchers reported having failed to reproduce another scientist’s work, and more than half had failed to reproduce their own.
A 400-paper audit of two leading AI conferences found that not one paper documented everything a reader would need to rebuild it. A cross-field survey traced data leakage through 294 papers across 17 scientific disciplines.
Most of the response to this has looked at the problem from the outside and after the fact: we run replication studies, audit how completely a corpus reports its methods, or bolt a checklist onto submission. All of that is useful. But it measures the outcome more than the mechanism. There is a complementary view that gets less attention – the one from the reviewer’s chair.
Over 2026 I have served as an invited peer reviewer across a range of engineering and computing journals, and what struck me was how little the trouble varied. The same small set of patterns turned up again and again. Peer review is confidential, so I treat that experience only as the lens that prompted this synthesis, not as evidence – everything below is anchored to the published literature and to a small pilot on public papers. But the patterns are consistent enough to name. There are, by my count, nine of them, and each carries a known, minimal remedy.
The nine failure modes
- Contradictory or non-traceable results. The simplest failure is a paper that disagrees with itself: the number in the abstract doesn’t match the table, or a headline gain in the conclusion appears nowhere in the results. When a paper won’t reconcile with itself, reproduction isn’t merely hard – it is undefined, because there is no single claim to reproduce.
- Incomplete methodological specification. A paper can be perfectly consistent and still impossible to run again, because it never quite says how – missing learning rates, batch sizes, seeds, splits, preprocessing. A method section can read beautifully and still describe an algorithm rather than the experiment that produced the numbers.
- Single-run reporting without variance. A great many papers give one number per method and crown a winner on a gap smaller than the noise of the training procedure itself. Without repeated runs and some measure of spread, a reader cannot tell a real improvement from a lucky random seed.
- Circular or self-referential evaluation. The “ground truth” a method is scored against turns out to be produced by the same pipeline the method relies on, so the evaluation was always going to flatter it. The numbers can look excellent and mean nothing, because nothing independent ever tested the claim.
- Data leakage. Test-set information seeps into training – features scaled on the whole dataset before the split, a predicted value sitting inside the input window, near-duplicate records on both sides of the line. This is the best-documented cause of results that look great and don’t hold up, and it is corrosive precisely because leaked results often do reproduce in the narrow sense: rerun the code and the inflated number comes back.
- Uncontrolled or strawman baselines. A method only means something against its alternatives. When every baseline is weak, borrowed from another paper on different data, or is really an ablation in disguise, the margin measures the baselines, not the method.
- Metric misuse. The wrong metric – or the right one stripped of context — hides the exact failure a reader needs to see: a single aggregate score that is unstable near zero, accuracy on imbalanced data with no precision or recall, a ranking on the very metric the method was tuned to.
- Overclaiming and scope inflation. Even a clean study can fail to reproduce its claims when the conclusion outruns the evidence – benchtop validation written up as if it were system-scale, a correlational design described in causal language. The results may replicate perfectly; the claim will not, because the claim was never the thing tested.
- Provenance and availability gaps. A paper can be consistent, fully specified, and still impossible to reproduce because the artefacts aren’t there – no code, no data, a dataset “available on request” that in practice is not, a silent dependence on a hosted model whose version is never stated.
The one distinction that matters most
If there is a single idea worth carrying away from this list, it is that these nine modes split cleanly into two kinds, and the split tells you what to do.
Some failures are cosmetic – repairable by editing the manuscript or depositing something that already exists. A contradictory number, a missing definition, an over-strong sentence, an absent availability statement: an author acting in good faith can fix these in revision.
Others are structural – undoing them means running the study again, not editing the text. Single-run claims, circular evaluation, leakage, weak baselines: these cannot be talked away in a rebuttal, because the problem is in the experiment, not the prose.
That distinction changes everything about timing. A structural problem is a cheap design change before submission and an expensive, often unanswerable objection after it. It is also what a reviewer most needs to communicate clearly: “please clarify” and “please re-run” are very different requests, and authors deserve to know which one they are receiving.
What the LLM era changes
The move toward large-language-model components – and toward using LLMs to help do the research – has not produced new failure modes. It has turned up the volume on the ones we already had. Nondeterminism gets worse, because many LLM pipelines are stochastic by default and hosted models can drift between opaque updates while keeping the same name. Provenance gets worse, because a result leaning on a hosted model is pinned to a version that can change or vanish. Specification gets worse, because the prompt is part of the method but is routinely paraphrased instead of quoted. And over-claiming gets worse, because fluent machine-assisted prose can make a thin claim read as authoritative – and can slip in citations that look plausible and turn out not to exist.
The remedies are correspondingly simple: pin exact model versions and snapshots, quote prompts verbatim, and check every reference.
Why this belongs to the whole scholarly-communication system
Reproducibility is, at root, an information problem — a question of how knowledge is recorded, described, and made re-usable. That places much of the remedy squarely in the hands of the people who steward the record.
Publishers and editors can set a short minimum-disclosure floor at submission and, for LLM-based work, require verbatim prompts and pinned model versions. A shared, detection-oriented vocabulary would also make reviewer reports more consistent – especially valuable as more venues experiment with open peer review.
Repositories and research-data services already curate the artefacts that the provenance mode depends on. Extending that with datasheet- and model-card-style documentation and FAIR-aligned deposit is exactly what turns “available on request” into genuine availability.
Librarians and data-literacy instruction can teach the earlier modes – leakage, circular evaluation, variance reporting – at the point where researchers are forming habits, which is far upstream of peer review.
None of this requires new machinery. It requires naming what goes wrong plainly enough that an author, a reviewer, an editor, or a curator can act on it in a single sitting. The reproducibility problem in AI research is mostly a reporting problem sitting on top of a smaller set of design problems. The same handful of failure modes come around again and again – and each one, once named, has a known and minimal cure.
The views expressed in this writing are my own. They have not been reviewed or approved by Tesla.
Mayank Vadaliya is an application support engineer at Tesla and a PhD candidate in information technology at the University of the Cumberlands