Algorithms help assign reviewers more quickly

Share this on social media:

Topic tags: 

Automating allocation of reviewers can speed up the review process - but the information about the reviewers' interests need to be good. Sian Harris speaks to the co-founder and CEO of open-access publisher Frontiers

Launching a new publisher with a new publishing model opens up opportunities to experiment with new approaches to publishing. This is what seven-year old Swiss open-access publisher Frontiers has been doing with its peer-review process.

‘When we started Frontiers we did it in the conventional way, with associate editors assigning reviewers but we found that it was a very lengthy process. It could easily take two months to invite reviewers because it is an iterative process and then we’d have to chase up to get the reports,’ said Kamila Markram, the company’s CEO and co-founder.

She recounted how her husband and Frontiers co-founder Henry Markram was an editor on the board of another journal where every time an article was submitted to that journal all of the board was informed and given the opportunity to review the paper. The board found this useful as a way to keep track of current research even if they were not interested in a particular paper, she noted.

Frontiers decided to adopt a similar approach with its review process. Each journal therefore has a significantly larger than usual board – ‘we really want to ensure that all the expertise is covered,’ she said – and everybody on the board is what the publisher calls a ‘review editor’. This means that initially they were all informed of all papers submitted and given the opportunity to be the reviewers.

‘Everybody on the board has been invited. They are all signed up, and so they should know about our approach and we are doing a lot of educate about the Frontiers process,’ Markram said, adding that this move was initially very popular with authors because, instead of up to two months to assign a reviewer, this process could be done within a few minutes.

However, this approach has not met with universal approval and the publisher was recently the subject of a post in librarian  Jeffrey Beall’s notorious Scholarly Open Access blog. Many of the comments and complaints raised in the post and the emails that Beall said he has received were about the company’s approach to peer review, in particular that researchers are asked to review papers that are not in their field.

However, Markram denies that the experiences shared in Beall’s post show a lack of quality in the process.

‘Peer review from our point of view is really at the heart of science. We have put in place a standardised review template that asks very detailed questions. We also publish the names of reviewers to make it transparent,’ she said. What has happened, however, is that the publisher has grown in size, particularly recently following a significant investment from Nature Publishing Group. This, she said, had unforeseen impacts on the publishing process.

The growth of the publisher has come with a growth in the number of submissions. This meant that the trickle of paper notifications to editorial boards that helped them keep up to date with the latest developments became a flood.

‘[The original approach] worked fantastically well for a while and then our journals grew. We became victims of our own success; the people who complained were those on our most successful journals,’ she said. For example, she noted that the journal Frontiers in Human Neuroscience now receives around a thousand submissions a year, which equates to a large number of emails.

The company therefore developed an algorithm to filter out relevant reviewers. This sends review invitations to 10 people, and then to 10 more if none of the first 10 are interested. ‘The algorithm is intended to accelerate the process and was built with authors and publication timing in mind,’ she said.

However, she admits that this is also not perfect. ‘We have put in place a system that matches reviewers with articles but the algorithm is only as good as the keywords that people put in,’ she explained.

‘When editors and reviewers sign up with us it’s very important that they fill in what they are interested in. This is also important for when editors assign reviewers manually to, which they can also do.’

She added that the publisher refines its review algorithm regularly in response to feedback.
‘Sometimes we get a bit of negative feedback. The burning feedback is always from people who are angry.

‘We are listening to what people are saying and modifying our algorithms on a weekly basis.'