Peer review beyond trust: why the system must evolve
Emerging challenges are exposing limits to a system that was not designed to withstand, writes Kim Eggleton
Peer review is at an inflection point.
For decades, the system has been built on trust in reviewers’ identity, independence and expertise. While editorial oversight has historically sustained this trust, emerging challenges are exposing limits to a system that was not originally designed to withstand. As a result, research quality assurance must evolve beyond a single checkpoint.
We need to embed rigorous checks across the entire research lifecycle and introduce systems that can detect patterns and risks beyond human capability.
Growing scale and scope of misconduct
The rise of reviewer fraud, particularly the emergence of “reviewer mills”, signals a shift from isolated misconduct to organised exploitation. These networks of compromised reviewers provide fake, generic or highly similar reviews for scientific papers often submitted under different identities with the goal to forcing authors to cite specific or unrelated articles to artificially boost citation metrics.
At the same time, pressure on the peer review system is also mounting due to the growing adoption of AI and large language models by researchers which has boosted the number of submitted papers. Not only has the number of papers submitted to journals taken flight since the introduction of ChatGPT and other generative AI tools, but it has also increased concerns around quality of scholarly work.
For example, a recent article in Nature highlights the growing presence of fabricated or “hallucinated” citations such as references that appear plausible but do not exist. How many of these hallucinated references are penetrating the scholarly record is not known but a study published in The Lancet uncovered a massive 12-fold surge in fabricated, AI-hallucinated references in the last three years. If references in articles are fabricated, what else is?
Citation manipulation, fraud and misconduct are no longer confined to the manuscript itself. We’ve all seen examples of editors and reviewers encouraging authors to cite their own work during the review process. These instances used to be limited to self-serving individuals and fairly easy to spot. They’re still there, but now we’re also seeing reviewers suggesting hallucinated references because they’re using LLMs to do their review.
What is even more worrying is that we see coordinated exploitation of the review process by paper mills, using positions on editorial boards, guest editorships and fake reviewer profiles to steer fake papers through to publication and insert references to whichever party has paid for them.
This reflects a wider shift: the mechanisms designed to safeguard research quality are themselves becoming targets for exploitation. This also shows why we need to be very careful about who we trust in positions of influence. Editorial board members need thorough vetting, and using author-recommended reviewers should be treated with extreme caution, because it can be open to abuse.
Detecting what trust alone cannot see
If trust is no longer enough, it must be supported by systems that can identify patterns beyond human checks.
IOP Publishing is tackling the challenge through the introduction of multiple systems, including Alchemist Review, an advanced AI-powered tool developed in collaboration with technology providers Hum and GroundedAI. It supports editors and reviewers by screening submissions for quality, fit and integrity, before they consume editorial time. With the tool, we support editors so they can focus their time where it matters most, and only send work out to reviewers that passes a range of human and technical checks.
We have also developed a home-built solution to help identify reviewer mills. The Duplicate Review Checker (DRC) applies machine learning to identify significant text overlap across reviewer reports. The tool is based on the principle that robust peer review should be uniquely commenting on the findings of the manuscript under assessment, as broader research shows that duplication in reviewer comments is highly unusual in legitimate peer review and can serve as a strong signal of coordinated misconduct. Duplicate reviews cannot be considered authentic, as quality peer review cannot be generically applied across different papers.
Developed and piloted in 2024, the tool has processed around half a million reviewer reports and now automatically checks all new IOPP submissions against this dataset.
The tool can identify cases where content has been substantially reused and pick up on patterns that would have been extremely difficult to detect manually. The tool is not meant or designed to replace human judgement, but it can enable earlier intervention and more focused investigations.
Rethinking checks and balances across the lifecycle
All of this requires us to look beyond peer review as a single checkpoint and instead consider the full research lifecycle.
Many Publishers, like IOP Publishing, already invest significantly in upfront integrity checks before manuscripts enter peer review. They screen for plagiarism, authorship anomalies, data concerns and other risk indicators, in an attempt not to burden the reviewer community with the influx of submissions.
This work is resource-intensive and continues to grow in scale and scope. But this layer of defence cannot sit in one place. Effective safeguards must be distributed and connected across the research lifecycle:
- Research environment and assessment: nurturing healthy research environments with scientific rigour and integrity as core values
- Research planning and funding: scrutiny of proposals and researcher track records, frameworks and support for healthy data provenance
- Submission and screening: structured integrity checks and editorial triage
- Peer review: independent expert evaluation, supported by verification and pattern-detection tools
- Post-publication: ongoing scrutiny, corrections and retractions
Each research stage, including publication, generates signals. However, one of the system’s current weaknesses is that those signals are not consistently shared.
Publishers, for example, may identify patterns or coordinated behaviours that could be highly relevant to institutions or funders. Yet there are limited mechanisms to feed this intelligence back upstream. Strengthening these feedback loops is becoming increasingly important to prevent fraudulent content creeping into the scholarly record at various stages of the publishing process, and worse, repeated patterns of behaviour going unchecked.
We have shared values and challenges yet despite coordinated attempts (such as United2Act) we seem to lack the ability to move forwards collectively at pace.
At the same time, we must be mindful of where responsibility sits. Authors are already navigating an increasingly complex publishing landscape. The challenge is to design systems that are robust behind the scenes, without creating friction for those acting in good faith.
A collective pivot
Peer review has adapted to challenges before. We’ve seen new models arise to fight bias and enhance transparency and new technologies to support better workflows. Now it faces mounting pressure from those looking to game the system. That said, it is good to acknowledge that the vast majority of researchers, reviewers and editors continue to act with great care and professionalism.
Our task is to ensure the system supports them while remaining resilient to bad actors. This requires us to pivot from trust as the default, to trust supported by verification, collaboration and provenance beyond peer review.
Kim Eggleton is Head of Peer Review and Research Integrity at IOP Publishing