Exposing peer review

When Nature launched a trial of open peer review in June 2006, its editors and publishers felt peer review was ‘never perfect’ and the time was right to ‘explore a more participative approach’. But minimal uptake meant that come the end of the trial, Nature director, Timo Hannay, had decided scientists ‘weren’t ready’ for open peer review yet.

A little over a decade later, a very different picture is emerging. The likes of BioMed Central, PeerJ and F1000 Research welcome this flavour of peer review while myriad journals, including Nature Communications, are trialling processes. And, importantly for scholarly publishing, industry heavyweight Elsevier has just revealed plans to add optional open peer review to its fleet of 1,800 journals by 2020.

‘We’ve had positive feedback from our pilot titles, so we now want to roll out open peer review across all Elsevier-owned journals,’ highlights Philippe Terheggen, managing director of the company’s journals group.

‘We are currently preparing the infrastructure for this, and will take a community-led approach with a three-year phased approach. It is best to have a multi-year phase in, and not a big bang, so this will take until 2019, and that is OK – we really want to take the market with us.’

It all started in January 2012, when the Elsevier journal Agricultural and Forest Metrology began publishing editor-selected peer review reports next to its published articles on ScienceDirect. Success prompted the ‘Publishing Peer Review pilot’ across four additional journals, to further test the waters on open peer review and get feedback on the thorny question, ‘Would you reveal your name in a published review report’? (see box out: ‘Elsevier’s radical idea’)

According to Terheggen, from the beginning reviewers were open to having their review report published, with nearly 50 per cent happy for the information analytics company to reveal his or her name. What’s more, editor and author interviews indicated that, of the reviewers that remained anonymous first time around, 36 per cent claimed to be happy to reveal their names in their next review.

‘Our pilot has included journals from very different fields and we’re just not seeing pushback from anywhere,’ says Terheggen. ‘And if I look beyond Elsevier, I see a relatively broad uptake from eLife to Nature Communications; there is no indication yet that a field is not suitable.’

For Terheggen, transparent and open peer review is critical to the future of journals publishing. For starters, he believes the process conveys trust among publishers, authors and reviewers: ‘If you are willing to share all information around peer review, this increases trust of already trustworthy information,’ he says. ‘And quality journals are happy to open up their peer review reports with consent from reviewers and authors, or course.’

What’s more, Elsevier surveys indicate open peer review actually increases the quality of review reports.

According to Terheggen, when authors taking part in the Publishing Peer Review Pilot were asked if they noticed an increase in the quality of review reports, a massive 70 per cent replied that the reports were more in-depth and constructive. As the managing director puts it: ‘This 70 per cent is a convincing number.’

To support such findings, the company is now rolling out open peer review metrics that, for example, measure how many reviews are conducted per article, how many reviews a reviewer has written, and also determine review report quality. Indeed, as Terheggen also points out, open peer review can bring much-needed recognition for the reviewer. Once a review report is made freely accessible and interlinked to the original articles, it is given a separate Digital Object Identifier. This means reviewers that choose to publish these reports with their names can claim the report as a publication and include it on their ORCID profile.

‘These review reports now become citable items,’ highlights Terheggen. ‘And reviewers are also telling us they will use their contribution as a reviewer on their CVs… honing your skills as a journal reviewer can help you to evaluate and write better funding proposals.’

Recognition counts

Elsevier is hardly alone in its acknowledgment that reviewer recognition is increasingly important. Launched in 2012, Publons is a website and free service for researchers to share, discuss and receive credit for peer review.

According to co-founder Andrew Preston, he and his business partner, Daniel Johnston, set out to improve peer review at a time when no obvious way existed for a reviewer to receive recognition. ‘It has always been hard to get recognition through review as most peer review was been either blind or unpublished, or both,’ explains Preston.

Given this, Publons has forged partnerships with Springer Nature, BMJ, SAGE, Wiley and more, and to date, more than 100,000 researchers have joined the site. As part of this, reviewers can choose whether or not to make review content open access following article publication, with Publons adhering to a journal’s review policy. But, as Preston says: ‘We do support all peer review processes, yet want to encourage open review, as we believe this is really important for trust, understanding and bringing context [to peer review].’

In its ongoing bid to promote quality review, Publons has devoted much effort to developing methods to measure the quantity of a reviewer’s peer review. However, in recent months, the company has also focused on measuring the quality of peer review.

According to Preston, editors managing a peer review process are asked to rate reviews on the basis of helpfulness and timeliness, with Publons then highlighting outstanding metrics.

‘Everybody loves this different concept of review quality and agrees it’s really important. We do hope that, as we scale up this process, providing more feedback to reviewers will improve the overall review process,’ said Preston.

But despite encouraging researchers to make peer reviews public, Preston hasn’t yet seen an obvious shift from traditional to open peer review. He points to a rise in interest and experiments, adding: ‘With increasing interest, I think the overall market will change over time but we haven’t seen this yet.’

‘One of the important things for us is to work with publishing partners and editors to educate them on the possibility of open peer review,’ he emphasises. ‘We need to show them that it isn’t something to be afraid of and that all the researchers publishing open reviews on Publons are having a good experience and that it’s also helping their careers.’

Rebecca Lawrence, managing director of life and clinical sciences publisher, F1000, agrees. Having launched open science publishing platform, F1000Research, in 2013, she says: ‘We were very familiar with closed peer review but have found it no more difficult to get referees for open peer review. I believe that this is because reviewers receive credit for review and they can cite their referee reports, as well as add it to their ORCID IDs,’ she adds.

F1000Research favours rapid publication followed by open, post-publication peer review. So, after passing in-house quality checks, an article is published and then referees are invited to review it, on the authors’ behalf. As Lawrence highlights, authors can suggest referees, but as she adds: ‘We’ve built this really powerful algorithm that looks at the article and its references to suggest referees. It locates senior authors in a particular field and also identifies conflicts of interest.’

According to the managing director, F1000Research referees often provide a detailed report that, as she highlights, can be very interesting in its own right. ‘These can be cited so people get credit which is such an important part of the publishing process that normally is completely invisible,’ she says. Crucially, for Lawrence, this peer review process is entirely transparent – with a referee’s name, affiliation, report status and review published alongside an article.

‘What we have found is that transparent peer review helps to remove a lot of potential conflicts of interest, as all sides have to declare conflicts publicly,’ she highlights. ‘We’ve had some instances where referees haven’t actually declared any conflict and other researchers have said “Hang on, this referee does have a conflict with the author”, so [transparency] does help to remove this.’

Last year F1000Research with The Wellcome Trust to launch Wellcome Open Science. The Wellcome Trust initiative operates on similar principles to F1000Research, which is testament to the publisher’s success, or at the very least, highlights peer review’s turning tides.

And, given the rising number of platforms and pilots – from Research Gate’s Open Review, and Peerage of Science to BioMedCentral’s Trials and PLOSONE – Lawrence is confident that open peer review is gathering momentum. What’s more, she is also optimistic that Elsevier’s strong interest in open peer review will spur industry action further.

‘This is great and the more [Elsevier] broadens out its [Publishing Peer Review pilot], the more open peer review will become normal, and researchers will become used to it,’ she says. ‘I see more and more pilots, and more and more publishers that are willing to go at least part way and test open peer review. And, as Elsevier has, these organisations are recognising that, actually, nothing bad happens and people quite like it.’

For its part, and given the scale of the task of rolling out open review across 1,800 titles, Elsevier is now focusing on making its peer review platform as straightforward as possible with technology playing an essential role in the process. For example, machine learning will link articles with appropriate reviewers. Meanwhile, software is under development as part of the company’s Reviewer Recognition Platform, which already provides a standard way of recording and acknowledging reviewer efforts in traditional peer review.

‘We are one of the first to be scaling up open peer review and others will now follow,’ says Terheggen. ‘I now expect that open peer review will largely become the industry standard, particularly if it is done in a way that allows people to opt in.’