Rogue peer review – a polysemy in the making

Lou Peck discovers that a challenging issue is also increasingly complex – but there is also plenty to be celebrated

It’s undeniable the huge investment that peer reviewers give to their community and how this effort is not only enormously valuable but helps us accelerate research. 

The rise in popularity of tools like Publons reveal the enormous effort of individuals in the community. I’ve been reading some recent author comments and it led me to explore the issue of ‘rogue peer reviewers’. What does that phrase even mean? Well, that’s a question. 

When I first started writing this piece, I thought I’d build on what I’d recently discovered and do some research to validate my thinking. A quick survey and a couple of interviews, I told myself, would do the trick – ‘bish bash bosh’. Well, let’s just say that the insights I collected led to more definitions making the polysemy ‘rogue peer reviewers’ even more complex than I originally thought.

I’ll start with the common explanations I knew:

• Someone who pretends to be a peer reviewer they are actually not;
• Someone who purposely slows down the process or outright rejects a paper to roadblock a publication so their work on a similar topic can be published first; or
• Someone who won’t review unless they can publish the article publicly online, e.g. in their blog.

At recent events, I wanted to hear from publishing representatives; the common recognised response was someone who pretends to be a peer reviewer when they are not. The general consensus was that they had not heard of someone purposely holding back reviewing work for their own gain, which is what I had started to hear from authors. This begs me to ask the question: is this on the increase and do we need to be mindful of it – or, after speaking to some editors who have heard of/experienced this, maybe it’s just not widespread enough yet? 

I took to Twitter with a survey to find out what I could discover, and wondered if I was about to open a can of worms. As well as those I already suggested, I found even more definitions:

• Someone who purposefully uses the peer review system to benefit themselves – taking manuscripts and presenting them as their own, not disclosing conflicts, etc;
• Someone who intentionally sabotages the work being reviewed;
• Someone who recommends a paper to be rejected unless it references the reviewer’s work; or
• Someone who writes negative reviews with no constructive content.

Now this research does not in any way discredit the phenomenal work that many peer reviewers do, but more address the recurring issues that are leaving a bad taste in the mouths of those involved – peer reviewers, authors, editors, or publishing staff.

It's amazing how one phrase can have different meanings to so many people. It’s worrying though, when you read the examples I was given; how do we address these and where do these challenges sit in the grand scheme of things? It’s a shame that the actions of a few can spoil it for the many but I guess the question on my lips here is, are the numbers of rogue peer reviewers on the increase? If so, what is contributing to this growth and how can we reduce or even eradicate this behaviour?

Peer reviewing is a hugely rewarding contribution to the community but sometimes feels like a thankless task with many asking for more recognition from commercial publishers whether rewards like APC’s waived for their next papers, monetary payment, etc or promotion and recognition from their employers. Some researchers are required to take on peer reviewing duties as part of their role and/or to help advance their career. That doesn’t mean it should be a difficult process. I know from my own explorations that publishers are supporting authors and peer reviewers to address these issues that are having a such negative impact – but some are clearly doing a much better job than others.

The survey yielded responders from a global audience, mostly from the academic sector – and I am quite sure that many would be of no surprise. Common challenges in addition to the definitions given earlier include: 

• Quality of peer review (language/references adjusted to peer reviewer style not journal style, meaningless and non-constructive feedback, negative review due to controversial content with no constructive criticism, lack of attention to detail by the reviewer giving feedback on something already discussed in the paper, recommended readings all by one person undermining reviewer’s anonymity, content critiqued for being too academic by an academic publisher, and quality reviews from real experts (not peripheral) within quick time scales);
• Quality of work (surprise at poorer quality papers from renowned authors, incomplete papers that need more work before being submitted and sent for peer review, assumption that English will be poor if not based in Europe or the US, and expectation that reviewers will edit grammatical errors);
• Rejection (paper rejected because reviewer doing similar work, rejected for minor typos, and rejected as peer reviewer is jealous and a competitive author)
• time (time taken to review, finding time to revise paper, unrealistic timelines, and patience required for non-Western authors to help them surface the underlying good research);
• Peer review recruitment (finding the best qualified people when asked to suggest reviewers by the publisher, continued engagement of quality peer reviewers, questionable practices by editors in how they recruit reviewers, inadequate quality reviewers, expecting chapter authors to review for other chapters in the same edited volume, continuing to keep peer reviewers engaged, work given to reviewers out of scope of their expertise, and blocking out the time of committed reviewers); and
• Industry pressures (reviewers passing work off as their own, continuing to support publishers as they lose money to open access publication and their profit models change, lack of trust in what gets published in open access, predatory publishers operating on a ‘pay to play’ model, competing bells and whistles to ensure that what is published is worthwhile research from respectable peer-reviewed sources, publishing only in institution recognized journals, not being able/allowed to publish negative results, having to publish papers as part of course completion e.g. degree programs, jealousy from competitive authors, and authors under immense pressure to publish by institution). 

These are real-life examples from a community of editors, librarians, peer reviewers and authors, and I’ll add the caveat that this was only a small survey to inform this editorial piece. What are we doing to address these challenges? We can of course look at revising existing processes and have better quality control measures, but what developments does the community want to see? 

Here’s what the survey respondents said:

• More rewards and recognition for peer reviewers;
• Employers recognising the peer reviewer contribution and applying towards a promotion;
• Wider adoption of peer review practices like transparent/open and double blind peer review;
• Authenticated peer reviewers – e.g. ‘fit for purpose’ assessment and email address verified;
• Constructive criticism;
• Faster time to publication – social sciences of note;
• Improved quality control and processes;
• More support and training for reviewers and editors to recognize and correct biases, and build papers with enough consistency to allow fast thorough peer review of suitably structured and novel work;
• Systems to help identify rogue behaviour and eradicate it;
• Authors struggling to publish good work helped with better ways of doing this; and
• Using a system that manages the whole process from submission to publication.

The industry is already taking steps to help address these developments and working on solutions that will better serve the system – but, with predatory journals on the increase, is this having an impact on peer review? One editor I spoke to said that they are regularly invited by predatory journals to submit papers by next week (literally), serve on their editorial boards, and edit issues.

We’ve still got some way to go and it will always need improvement, but many publishers already have fantastic resources to support and train authors and peer reviewers. I just don’t think people know what information they have available freely at their fingertips. I’ve started to build my own central URL repository of these resources and always welcome any additions – what’s the point of reinventing the wheel when someone has already done a brilliant job?

F1000 has an open peer review platform that I’ve used and found it works well – just because I knew the reviewers, didn’t mean they were biased! It’s highly likely you will know them from your field of expertise, or at least by reputation.

Thieme piloted crowdsourcing peer review (intelligent crowd review) with their Synlett journal and yielded really interesting results – more than 100 qualified reviewers commented on one paper in 72 hours.  Several other publishers are now trialling this approach – this innovative why of peer review will present its own challenges, but hopefully we’ll work together to learn from past mistakes and build a more robust, fair workflow. Wiley is another great example of a publisher training and support for their editors at their editor annual conference.

Of course it doesn’t all fall on the shoulders of the publishers and editorial boards to improve peer review, but to the community as a whole. Authors should feel encouraged to report if they feel a peer reviewer or editor is acting unethically, or to note non-preferred reviewers if identifiable with their manuscript submission. If reviews aren’t constructive, make sure this is fed back – certainly some editors I have spoken to actively reject reviews and get another for the author if it is not constructive.

Some of these challenges are not going to be addressed overnight, but I’d like to see more intensive research done in this area encompassing a whole range of publishers/service providers and the author and peer review communities, concluded with a clear strategy and recommendations. What trends are we seeing, how common are these challenges, what is being done to reduce or even eradicate these issues, and how can we collaborate through our learned experiences to support this community?  

Let’s celebrate our community by supporting them further in what they do and start addressing some of these issues before they become significant problems – if they aren’t already.

Lou Peck is the founder of publishing and marketing consultancy The International Bunch