Peer review, preprints and a pandemic

Even before the Covid-19 pandemic struck in 2020, staff at MIT Press were noticing a problem with preprint servers. Over time, more and more preprints were being published and drifting into mainstream media, even government, in ways that weren't always helpful and were sometimes even misleading.

Then came coronavirus. As Nick Lindsay, director of journals and open access at MIT Press, puts it: 'These issues were exacerbated as the sheer volume of research we were seeing on bioRxiv, medRxiv and other preprint servers was immense. Literally thousands of preprints were going out there with no review, and we started to see some really troubling things take place.'

Amid the torrent of data released onto preprint servers, research clangers emerged and withdrawals, retractions and expressions of concern followed. For example, in late January, 2020, a bioRXiv preprint from a group of researchers from the India Institute of Technology reported HIV insertions in the spike of SARS-CoV-2 that were not present in past coronaviruses. The researchers also speculated these had been placed in the virus intentionally. Then around a week later, a ResearchGate preprint from a researcher at the South China University of Technology and colleague, proposed that coronavirus 'probably originated from a laboratory'.

In each case, the papers were re-drawn following outrage from the research community. Chinese government and World Health Organisation have since condemned such reports, but these now infamous publications undoubtedly fuelled the already widely-circulating coronavirus conspiracy theories at the time. Herein lies the problem with preprints.

'With such examples in mind, myself and Amy Brand [MIT Press director] put our heads together and asked, how can we help here?' says Lindsay. 'As we talked, it became clear that this was a big opportunity to have a positive effect on the public understanding and trust in Science and also offer a service to mainstream media, researchers, scholars and clinicians that needed a preprint verification that wasn't yet happening.'

Rapid Reviews: COVID-19 (RR:C19), headed up by Professor Stefano Bertozzi from Public Health at the the University of California Berkeley, quickly followed. Described as an 'open-access overlay journal', the publication aims to accelerate the peer review of Covid-19-related research preprints to advance findings and prevent the dissemination of false or misleading news.

To speed up the process, the editorial team, including an army of graduate students, selects potential Covid-19 preprints for review, from preprint servers such as medRxiv, bioRxiv, SSRN, with help from 'Covid Scholar'. This text-mining tool was developed by Berkeley Lab materials scientists to help researchers wade through the mountain of Covid-19 literature.
The chosen preprints are then sent out to RR:C19's pool of reviewers, who will answer key questions such as is this preprint reliable and trustworthy, should it be taken seriously, how might it be used to further our knowledge in fighting the pandemic?

As Lindsay highlights, this isn't traditional peer review. RR:C19 is trying to balance the need for rigour with rapid response, and as such, preprints should be published, with two finished reviews, within two weeks.

'Since we launched RR:C19, we've seen that the preprint servers link [preprints] back to our reviews - so they clearly see a need for review,' highlights Lindsay. 'And publications such as the New York Times are also picking up our reviews and using them as evidence.'

For Lindsay and many in scholarly publishing, this wider understanding of the preprint is critical. As he puts it: 'Faculty understand the difference between a preprint and a published article, but for plenty of others it's difficult to understand what the difference actually is... We need to engage in efforts to ensure people understand that there is a profound difference between the preprint and published article.'

Nic Marsh, senior researcher at The Peace Research Institute Oslo (PRIO), agrees. Since the onset of the pandemic, Marsh has also noted how more and more preprints are being used more broadly than ever before. 'The Public doesn't always know the difference between a peer-reviewed journal and a predatory journal or even a preprint server - if it looks like academic research then it can be really difficult for someone to spot what's high quality and what isn't,' he says.

But, as Marsh also points out, pandemic-related preprints have also been more widely used to further research: 'I've seen some really significant findings first published as a preprint and quite senior researchers using this a route to publication. It is important to note that research not published in academic journals is commonly reviewed by peers for publications – and such non-anonymous reviews can be very useful.

Understanding change

Researchers at PRIO seek to understand the processes that bring societies together or split them apart, and in recent months Marsh has been investigating peer review and the societal impact of different forms of Covid-19 research publications. His study, based on a Covid-19 Reddit forum comprising some 300,000 members, indicates that preprints and other non-peer-reviewed publications, such as press releases, are widely read and are now challenging peer-reviewed publications as a means of disseminating research.

'With the global pandemic there's a clear need to get information out as quickly as possible,' he says. 'This underlines that even though many publishers have accelerated the publication process, it's still perhaps too slow for a pandemic.'

Clearly ongoing debate on the role of peer review in the research lifecycle will continue to hold importance for many years to come. But for now, will the likes of RR:C-19 and similar endeavours such as the rapid review of Covid-19 Registered Report submissions by PeerJ, PLOS Biology and other journals, help to deliver the current need for speed?  Professor Detlef Weigel, director of the Max Planck Institute for Developmental Biology and eLife deputy editor, thinks so.

In late 2020, eLife announced that come July 2021, it will only review manuscripts already published as preprints, and will focus its editorial process on producing public reviews to be posted alongside the preprints. The open access publisher's new 'publish, then review' model follows in-house analysis that indicated around 70 per cent of papers under review were already available as preprints.

As Weigel says: 'I wouldn't say our move to this model is tied to the pandemic, but certainly the vast volumes of Covid-19 papers appearing on [preprint servers] have pushed us towards this.'

Right now, eLife is phasing in public review with authors currently retaining a degree of control over when the review is published. If editors decide a paper is not appropriate for eLife, they will allow authors to postpone the posting of the public review until the paper is published elsewhere, so unfavourable review will not influence eventual publication. Weigel anticipates that within the next 'two years or so', a public preprint review will become the default. And in line with RR-C19's New York Times experience, he adds: 'By allowing reviews to be attached to your preprint you can show a journalist that your work has been reviewed and there's a higher chance that the world will take notice.'

In the meantime, eLife has been instructing its editors and reviewers to write reviews for a public audience. 'We've been working very hard on our reviews – reviews for the public need to be written in a different way than those for just the author,' says Weigel. 'New guidelines for review start with an evaluation summary... these are going to be so much easier for someone from outside of research to understand.'

Weigel also reckons preprint review will be significantly faster than traditional peer review – and he is certain review quality will not suffer. 'We have a large cadre of senior editors who are committed as well as amazing staff... not all journals have this luxury, which is why we can afford to be bolder,' he says. 'I think initially we will need to work with reviewers to do things differently, but once other journals see that this works well then habits are going to change.'

With 'publish then review' in place, Weigel and eLife colleagues hope to eventually create a system of curation around preprints that replaces journal titles as the primary indicator of perceived research quality. 'Of course we believe in peer review and for a while we hope that our [publish then review] model will co-exist with traditional peer review,” says Weigel. “But in 20 years I believe this will all move to the publish then review and curating model,' he adds. 'It's down to culture change and the new generation of scientists, students and postdoctoral researchers are getting used to putting their research onto preprint servers.'

But culture change aside, who will pay for these emerging models of preprint review? While MIT Press's Lindsey is keen to extend preprint review to other fields, primarily climate change, he does wonder where future funds will come from. 'The Patrick J McGovern Foundation has been incredibly generous but we are still working on a sustainable business model that will allow us to keep Rapid Reviews going,' he says.

And eLife finds itself in a similar predicament - the publisher is a not-for-profit business but has expenses.

'How to make money out of all of this really is the elephant in the room,' says Weigel. 'Funders and libraries might say, "these organisations are producing something valuable and we as a community are willing to pay for it" but it's a big hurdle.'

Tradition and transparency

Still, for many in the world of peer review, tradition largely remains, at least for now. In September 2020, the Institute of Physics Publishing (IOPP) laid out plans to move all of its owned journals to double anonymous peer review – where the reviewer and author identities are concealed – by the end of 2021.

The decision is intended to tackle gender, racial and geographical under-representation in scholarly publishing, and follows positive trials on journals that involved single-anonymous and double-anonymous peer review. IOPP also offers transparent peer review on a number of its journals which gives authors and reviewers the option of publishing an article's peer review content in a discoverable and citable form.

'We're not forcing anyone down this route but we felt it was really important to give authors and reviewers the choice to display the review history of the article and have as much transparency in the process as possible,' highlights Marc Gillett, head of publishing operations at IOPP.

At the same time, the publisher is intent on driving efficiencies across its peer review process. As Gillett puts it: 'We consistently hear back from authors that the speed of peer review is one of the top things that they pay attention to when considering which journals to submit to.'

Given this, IOPP has launched a training and certification programme to support researchers in peer review and has also diversified its reviewer pool to tackle the well-worn issue of reviewer fatigue. However, the rise in preprints is undeniable – physicists have long-published research on preprint server, arXiv - so with this in mind, the publisher has also started trialling an option for authors to list and link to their preprints while a manuscript is under peer review at the journal.

And undoubtedly like many, Gillett is also watching eLife's latest 'publish, then review' model with interest. 'It is possible that the nature of peer review will change over the next couple of years as we find new ways for preprint platforms and journals to complement each other, but it's really important that research undergoes some form of peer review,' he says. 'Whether it's a preprint or accepted manuscript, researchers ultimately need to have a trusted method of quality assurance and this is something that we as publishers can deliver.'

So, in the time of Covid-19, what next for peer review? Trust was the theme of Peer Review Week 2020, which for Lou Peck, 2020 steering committee co-chair from scholarly communications specialist consultancy The International Bunch, will remain as important as ever. 'The peer review process is built on trust,' she says. 'Trust in those submitting articles, trust in the peer reviewers, trust in the process itself, and the organisations behind it, and finally trust in the published output.'

However, Peck also reckons that the pandemic has pushed peer review, and the quality of published work, to the forefront of many people's minds. 'One significant consequence of Covid-19 is that it has highlighted the impact of bad science and fake news, and how valuable peer review is. Publishers, service providers and those in research support have proactively taken steps to offer more support around peer review,' she says.

'I believe we've experienced more of a stakeholder community-driven approach, and as a result, I hope we continue to see more collaborative working, as ultimately, we are better together,' she adds.

MIT Press' Lindsay concurs, but also believes the pandemic has truly underlined the importance of transparency in peer review. 'Many editorial offices see the anonymity of peer review as something that needs to be preserved in order for people to be forthright and open,' he says. 'But I think those arguments have been outweighed by this need to get quickly legitimate research to other researchers and clinicians. Fingers crossed - we're now looking at an era of more transparent and open peer review.'