Gender, geography and seniority

Kim Eggleton wonders: how do we solve the problem of diversity in peer review?

Opportunities to publish scientific discoveries should be open to all, subject to fair peer review. But there is growing evidence the peer review system is not as fair as it could be, and that publishers need to improve equity of opportunity.  

Theoretically anyone can submit to any journal, but are all submissions assessed in the same way? How can publishers ensure consistent and fair peer review? After all, every manuscript is different and peer review is essentially opinion. So how do we tell if manuscripts are being judged on merit alone? And if there is a problem, what can we do about it? 

Equality goes hand in hand with diversity. Countless studies show diversity of thought leads to better science, so surely all publishers should be aiming for contributor diversity? We should ask ourselves: is everyone who submits treated the same, and getting the same opportunities? Is everyone represented, and in the right proportions? These are the questions many publishers, including IOP Publishing, are trying to answer. 

What does diversity in peer review look like?

The demographics of journal submissions have shifted over time.  

For example, between 2014 and 2019, IOPP’s submissions from India doubled. But submissions from the USA grew by just 6 per cent. Accepted articles saw similar changes – phenomenal growth in published work from India and China, yet very little growth in accepted work from regions that used to dominate publication output. This no doubt reflects the increased investment some  countries are putting into R&D and education, as well as the increasingly global pressure to publish in high ranking journals. 

One could expect that as the demographic of both submissions and published work has shifted, so has the demographic of those conducting the peer review. On analysing our own data, we (and likely many other publishers) realised our reviewer pools and editorial boards lagged in terms of geographic and gender representation. Long story short: we have not kept up with the times. And we need to do something about it. 

If the demographics of invited reviewers broadly reflected those of published authors, we should have expected around 27 per cent of invited reviewers to be from China and 11 per cent to be from the USA in 2019. But in reality, only 12.5 per cent of reviewers invited were from China in 2019, compared to 23 per cent from the USA. That is a huge and worrying difference. Why? Because evidence suggests there’s geographical bias in peer review. 

‘Reviewers were also more likely to accept invitations to review articles when the corresponding author was from their region and were more likely to be positive about such articles’ (Gaston & Smart, 2018)

‘US reviewers recommended acceptance of papers submitted by US authors more often than did non-US reviewers’ (Link, 1998). If these studies are correct, the disparity between reviewer and author demographics suggests authors from reviewer-under-represented countries may be disadvantaged, especially under the single-blind review model all our journals operate (some offer a double-blind option in addition). 

This problem is not unique to IOPP. The Global State of Peer Review report – a study across hundreds of journals and publishers – says: ‘Established regions review more than emerging regions relative to their respective article outputs’. This is supported by looking at the chances of a reviewer from these regions accepting an invitation to review. Our data shows a reviewer from China is more likely to accept a request than one from the USA. A reviewer from India is even more likely to accept.

Looking at career stage shows similar results. The more senior the researcher, the more likely they are to be asked and the more likely they are to decline. This has long been the presumption, but it’s striking to see in black and white and it’s concerning. It explains the increasing difficulty in finding reviewers – demand is growing at an incredible rate and the same people (senior academics from mostly Western regions) are being asked time and time again to review. As a group, their numbers aren’t increasing enough to keep up, so they’re fatigued.

Sadly, it’s a similar story with gender balance. Proportions of submissions from women consistently outweigh those of women invited to review manuscripts. Representation on editorial boards is even worse, with some as low as 3 per cent. 

Admittedly, we publish in disciplines with poor gender balance, but even so the figures are depressing. Some causes are quite easy to guess; for example, editorial boards mostly consist of senior academics, and fewer women reach these levels due to the ‘leaky pipeline’.  But could there be other factors at play? This under-representation of women seems to be the case for other publishers too; The Lancet, Nature and the Royal Society of Chemistry have all made public statements about the lack of gender diversity in their journals, and committed to improve.

The causes for the disparity in reviewer selection are, we believe, multiple and complex. At IOP Publishing we manage the peer review administration on behalf of journal editorial boards, including reviewer selection. Evidence from our editorial teams suggests lack of reviewing credentials puts our in-house editors off using potential reviewers. Although our database has grown with new authors from China, many are early career researchers with little or no reviewing history.  

This, alongside our editors’ desire to approach senior academics as reviewers, helps explain some of lack of diversity in reviewer selection. Identifying reasons for the gender disparity is harder. We suspect though, that the  lack of diversity on our journal editorial boards is also playing a part. Editorial board members often suggest reviewers, and if they mostly recommend reviewers ‘in their own image’, (as research suggests) it compounds the problem.  

What are we doing to try and improve the diversity of our reviewer pool?

While the current reviewer data in isolation is worrying, we are seeing reviewing activity growing quite rapidly year on year in countries like China and India. But rather than simply relying on that underlying organic growth, we wanted to take a proactive approach to:

• Diversifying editorial boards: we know our editorial boards are not geographically reflective of our author bases on many of our journals, and they don’t reflect the gender balance either. We’re addressing this by broadening our networks and paying attention to recruiting from under-represented countries. To do that, we’re spending more time and money on improved data, and getting to know the research communities in those countries better. We’re also trying to recruit more women onto our editorial boards, although we recognise the real work needs to be done in encouraging more girls and women into STEM education and employment. IOP Publishing’s learned society status means that any surplus we make from our publishing activities goes into promoting physics for the benefit of all. 
• Educating our editorial staff: all our editorial staff receive training on unconscious bias in peer review when they join the company, and refreshers are offered frequently. We also encourage editorial staff to be much bolder in their reviewer selection. Giving them data helps. For example, they now understand that reviewers from China and India are more likely to accept an invitation to review, and will also do it faster. Some journals are considering a quota approach to address the disparity in reviewer invitations and reviewer fatigue. We also offer diversity webinars for our editorial board members. We held the first in March 2020 and more than 100 board members joined, with great levels of engagement. In the future, we will run more of these, and offer them to our reviewers too. 
• Building our reviewer pool through co-review: we are introducing a formal method of co-review, initially on three journals, to help early career researchers build their reviewing history and reputation. We know reviews are often delegated to junior researchers (McDowell et al., 2019). But because this is done informally, the established reviewer’s records grow, and the junior researcher goes unrecognised. By formally acknowledging when co-review occurs we give credit where it’s due, which should increase the numbers of review invitations early career researchers receive. 

What are we doing to improve equity of opportunity?

All these steps, if successful, will help improve consistency and fairness in assessment. But we can, and will, do more. 

We’ve started offering transparent peer review on three of our journals, so readers can see the standards of peer review applied and decide for themselves if we’ve made the right decisions. We’ve been offering a double-blind option on a handful of our journals for a few years, and despite the modest take-up of ~20 per cent of authors, we know (through our research and the wider literature) double-blind is perceived as an effective way to reduce potential bias.

We’re going to change how we ask reviewers to rate and score manuscripts (with better instructions), making it easier for them and our editorial teams to be objective. We’ve already improved our reviewer guidance, so reviewers are informed about implicit bias and what they can do to counter it. We’ve produced a code of conduct for board members and have clear guidance on our website about our stance against discrimination. We expect authors, reviewers, board members and staff to treat everyone with respect, and to judge work on its merits alone.

Finally, we’re being open and honest about the challenges. We know we’re not alone in these issues, and we hope by acknowledging the problem and openly discussing possible solutions, the whole scientific community can benefit.

Kim Eggleton is research integrity manager at IOP Publishing