Peer review under pressure?

Daniel Johnston, of Publons, says academic editors are coming under increased pressure to find over-worked researchers willing to get involved in the peer review process

We are all starting to realise it’s unsustainable to continually ask time-pressured researchers to do peer review when there is no recognition or incentives for doing so. Without any sort of reward, peer review is seen by many as a chore distracting from more career-relevant work. The best reviewers are over-worked and decline many more review invitations than they accept, leading busy editors to gamble on unknown and unsuitable reviewers. The rise of article retractions and cases of peer review fraud in recent years is a sad consequence of this.

On a more positive note, there are a couple of really promising trends in the industry that provide hope: the drive for greater transparency in peer review (eg. Frontiers, PeerJ, GigaScience), and the desire for researchers to record and get credit for all of their research outputs (eg Figshare, ImpactStory).

Publons sits at the nexus of these two trends, enabling reviewers to turn their peer review contributions into something they can put on their resume and bringing transparency to peer review – while still allowing for reviewer anonymity. The hypothesis is that incentivised reviewers and greater transparency results in faster, more efficient, and more effective peer review.

One big realisation is just how important the role of an academic editor is. They play such a vital role in organising and monitoring the peer review process, but much of the talk of ‘fixing peer review’ neglects to even mention editors.

We’ve devoted a lot of time in recent months to developing tools to help editors to better perform this vital role; tools specifically designed to help editors find, screen, and reward legitimate and suitable peer reviewers. And, as with reviewers, we provide a way for editors to get credit for all their editorial work too.

Going forward, the most significant development will be the drive for greater transparency in peer review.  Publishers are looking for ways to thank their reviewers and prove that their journals are doing a good job of peer review without uprooting their long-established blind or double-blind peer review culture. We’re working with publishers to help with that.

Scamming is a very serious issue that is causing a lot of headaches for publishers. There have been upwards of 100 retractions due to peer review fraud in the last year or so, and we’re likely to see even more cases come to light this year.

It’s a difficult problem to solve. Without the time, the expertise, or the tools to properly screen invited peer reviewers, editors are in a really difficult spot. They’re asked to find between two and 10 peer reviewers per manuscript, and for each of these to check that the reviewer is not: a fake name with a fake email; a real name with a fake email; a colleague, coauthor, or relative of the author(s); or otherwise unqualified to review the manuscript.  Those are tough questions to answer.

For our part, we’ve built a reviewer screening tool that utilises review history and verified email address information from the world’s largest cross-publisher reviewer database to allow editors to confirm the legitimacy and suitability of potential peer reviewers. For instance, we check the reviewer name and email address against our database and if a supplied email address was used as the corresponding email address in a trustworthy academic journal, we can provide the editor some peace of mind as to the reviewer’s legitimacy and review history.

Better in-house fraud detection by publishers will certainly help, but it is our opinion that the ultimate solution to the peer review fraud issue lies in an independent database of verified reviewers and journals, which is what we’re working with publishers and reviewers to establish.

Janne Seppanen, managing director of Finnish company Peerage of Science (pictured above), is concerned over the growth of ‘trash, predatory, fraudulent, and vanity journals’

From my perspective the biggest and most important change is the wider-scale publisher acceptance or at least serious consideration of the concepts behind Peerage of Science. Open engagement from a pool of validated scientists (note: NOT open peer review), cross-reviewer evaluation of review quality, and concurrent or at least sequentially shared use of peer reviews among journals are ideas that only three years ago were just challenging the old system, now they are changing it.

Then there are negative developments that will have huge impact in coming years.

First, the growing enroachment of trash, predatory, fraudulent, and vanity journals, which claim to do peer review but do not. Second, the revelation that both the management and execution of peer review in even the most prestigious journals and in well-established publishers is too often careless. Examples are the recent revelation that purchased research papers from Chinese ‘paper mills’ have routinely made their way into respected journals; the fact that in several cases and in respected journals, authors have managed to become reviewers for their own manuscripts under a generic email address and faked identity; and the infamous ‘arsenic life’ and ‘STEM-STAP’ cases where peer review in the paragon journals failed to do what clearly was within the core purpose and reach of the process.

These developments will soon lead to a situation where the world needs better, objective, independent and trustworthy definition of ‘peer reviewed journal’. Simply stating that you are one is no longer nowhere near enough. Nor is the presence of distinguished editorial board. Nor is being owned or operated by a major publisher.

You need to convince the world that you only publish things if they meet your stated criteria in peer review. In this regard, I feel the ‘all sound science’ journals disparaged by their critics to be doing ‘peer review light’ are actually doing their jobs better than the established order does.
Without these developments, we might have been tempted to alter course at some point, to incentivise quicker, more lenient and ‘collaborative’ peer review to attract more authors. But seeing how badly the old system fails in serving the higher ideals of science, reinforces my conviction that we are on the right track though manuscript submission rate has not grown as fast as we would have hoped.

Scamming is a big problem. One solution would be to have enough full-time editors, who focus on making editorial decisions, rather than in being  managers of peer review who delegate their powers to the ‘third reviewer’. The editor should always have time to read the entire manuscript.
Another thing is that, although I sincerely endorse the ideal of editorship, and want to help enhance their role, the editor should no longer be beyond scrutiny. There is much talk about the merits or problems of double-blind versus single-blind versus fully disclosed peer review, but why does no one ever even mention the fact that the deciding editor always knows author and reviewer identity and affiliations? How come maintaining objectivity and avoiding bias is an issue for everyone, except the one person who has the most power in the transaction? With great power should come great responsibility.

Alison O’ Connell, marketing manager at Aries Systems, based in Massachusetts, USA, says she wants the academic community to be inspired to adopt standards and deploy editorial software solutions

The most important developmental changes within peer review, and the scholarly publishing ecosystem as a whole, are the emergence of viable data standards – including ORCID for disambiguating identities, Ringgold for institutional name normalisation, FundRef for accurate metadata capture of funders IDs, and JATS for standardising the XML tags we all use – enabling systems integration, efficient machine to machine workflows and reliable data rendering.

The implementation and adoption of these standards allows Aries to develop software solutions that are applicable to more journals. For example, working with the Copyright Clearance Center (CCC) we have developed an API that allows for the transfer of standardised data to enable efficient processing of APCs (article publication charges). In other cases, the adoption of standards has resulted in improved author and co-author experiences – for example, Single Sign On with ORCID, so that authors submitting to more than one participating journal don’t need to remember several different logins to submit across sites.

The adoption of standards and APIs has allowed Aries to implement solutions that better serve the 6,000 journals using our Editorial Manager system. We’ve also seen accelerated adoption of our production tracking module by publishers such as Wiley VCH and the Society for General Microbiology. We deliver three highly functional releases per year, and by integrating standards, there’s been no shortage of opportunity.

The challenge for us is inspiring the community to adopt and deploy these new solutions. There is so much simplification that can happen, but only if the functionality is switched on.
To that end, our challenge is persuading publishers to adopt standards and configure the solutions that will prove efficient instantly.

Scamming is an industry problem that we take very seriously. In collaboration with customers, Aries is reviewing technologies and protocols that could be deployed so as to increase the confidence level that users are who they claim to be.

Jody Plank, operations manager at North Carolina-based Rubriq, says pressures on peer reviewers suggests a need for them to be validated by the industry

A significant development is the continued rise of broad-scope, novelty-agnostic journals in the industry. This trend has created a new form of peer review, as the reviewers for these journals are no longer asked to judge the potential impact of a manuscript when assessing thepublishability of the work.

This sounds like a very subtle thing to many of us, but attempting to match the novelty and interest of the work to the journal considering it for publication has become so ingrained in the peer review process that some reviewers have difficulty evaluating a manuscript without consideration of the impact. However, it is more fair to the authors to use the experts in the field to evaluate the soundness of the research and allow the field as a whole to determine the actual impact of the work instead of asking just a few to predict that impact and have the publication of the work based on that prediction.

The idea of evaluating the execution and presentation of the research separately from the potential impact of the work has shaped Rubriq from the beginning. Our scorecards allow the reviewers to comment on these aspects in separate sections of the review, and the scores for these sections can be evaluated separately by authors and journal editors.

In addition, the number and universal criteria that the broad-scope, novelty-agnostic journals use for publishing research has enabled us to provide Sound Research Stamps, which will help researchers understand whether their work is appropriate for submission to a growing number of outlets.

Of course, it also allows a large number of journal editors to understand when a Rubriq reviewed manuscript would be a great fit for their journal.

The massive scaling of the publishing industry both in terms of the number of manuscripts and the number of journals represents one of the largest challenges to the peer review community. Reviewers are receiving more and more requests to review manuscripts, and the best reviewers are in high demand. This is creating a challenge as researchers are now having to spend a significant amount of time managing their peer review commitments in addition to the time required to actually perform the reviews.

The size of the research community has created an environment where editors cannot always know every active researcher in a field, and the nature of electronic communication makes it easier for researchers to be impersonated by others.

When combined with the time pressures that can exist within the publishing industry, the publishing pressures within academia, and the history of the industry to function on ‘the honour system’, these factors have created a fertile environment for these fraudulent reviews and the publications that have resulted from them.

Because of this issue, Rubriq independently vets each of the reviewers that work with us rather than simply accepting credentials or a user-generated email addresses.

Kaveh Bazargan of River Valley Technologies has developed its own ‘intuitive’ peer review platform to compete with current systems

The market doesn’t seem to have changed a lot, and is not taking advantage of ‘Web 2.0’ technologies, so is not really providing for the more progressive publishers. Submission and peer review is still a chore for all concerned, not a pleasure!

Our background has been in typesetting and composition work, so we have not been a player in the peer review market. However, having seen the challenges and costs publishers are facing with current systems on offer, we decided to build our own intuitive peer review platform, ReView.

Our system doesn’t require any installation by publisher and in fact it is a single installation for all journals in a publishing house. This means, for example, that manuscripts can be moved from one journal to another painlessly, and that reports can include data globally across all journals. This is how we see the future of peer review.

In 2014, we launched our end-to-end publishing platform. We see this as the holy grail of publishing. If authors write online in our platform, their work will be automatically structured and saved as XML on the fly. The next stage of the platform is peer review, where reviewers make comments in the very same document, hosted in the cloud. This obviates the need for files being emailed back and forth.

Once the submission is accepted, copyeditors, journal editors and authors can also make comments and changes directly in the XML. Once all changes have been accepted, the publisher, author or typesetter can create the desired output PDF or ePub, etc., on the fly through our RVFormatter platform, or manually in their traditional workflow if they prefer. We see this as the future of publishing, drastically reducing turn around times and processing costs, and even moving to immediate publication. We estimate up to 90 per cent reduction in current processing costs of manuscripts.

The root of the problem in the recent ‘scamming’ stories is the pressure put on academics to publish in high-impact journals. This is pushing academics to somehow ‘game’ the system. Apparently there are now even companies who offer the complete service including authoring and peer review to prospective authors. The mandatory incorporation of ORCID for authors will go a long way towards preventing such misuse.