Scepticism over increased use of AI in research assessment

There is “widespread scepticism” among academics and professionals over the use of AI to assess the quality of research, according to a new report.

The report, led by the University of Bristol and funded by Research England, shows that some universities are already using generative AI for this purpose – but that there is a need for “national oversight and governance”.

The UK’s system for assessing the quality of research in higher education institutions is known as The Research Excellence Framework (REF). Its outcomes influence how around £2 billion annually of public funding is allocated for universities’ research.

The last REF took place in 2021 and changes to guidance for the next one – REF2029 – are expected to be announced this month. Total costs of REF2021 are estimated to be around £471million, with an average of £3million per participating higher education institution, and REF2029 finances are anticipated to be much higher.

Richard Watermeyer, Professor of Higher Education at the University of Bristol and lead author on the report, said: “GenAI could be a game-changer for national-level research assessment, helping to create a more efficient and equitable playing field. Although there is a lot of vocal opposition to the incorporation of it into the REF, our report uncovers how GenAI tools are nevertheless being widely – if currently, quietly – used, and that expectation of their use by REF panellists is high.

“The report is timely given the immense financial pressures facing the sector. It’s widely accepted that the regulatory burden of the REF is high and will, in all likelihood, only increase. Our report demonstrates that GenAI has the potential to alleviate some of this, but offers no complete solution. It could also create new bureaucratic challenges of its own, including establishing new requirements and protocols for its appropriate use.”

The report investigated the usage of GenAI at 16 HEIs, including Russell Group universities and more recently established universities, across the UK. Findings indicated evidence of GenAI being widely deployed to prepare REF submissions in some capacity. But the extent and way it was being used greatly differed. The study also included a survey of nearly 400 academics and professional services staff, which asked how they felt about GenAI tools being used for various aspects of REF2029.

For all aspects the majority of academics and professional services staff were shown to strongly disagree, with the level of strong opposition varying between 54% and 75% of respondents for different parts of the REF process. There was most support for GenAI tools being deployed, among almost a quarter (23%) of respondents, to support universities in the development of impact case studies.

The report makes a host of recommendations, including that all universities should establish and publish a policy on the use of GenAI for research purposes, encompassing REF; relevant staff should receive full training on the responsible and effective use of AI tools; and appropriate security and risk management measures should be implemented.

It also calls for robust national oversight, comprising sector-wide guidance on usage for REF29 and a comprehensive REF AI Governance Framework. To help achieve equitable access to the technology among all HEIs, it advises that a shared, high-quality AI platform for REF could be developed and made accessible to all institutions.