Can AI create high-quality, publishable research articles?

Niki Scaplehorn, Springer Nature

Niki Scaplehorn reflects on a three-day ‘hackathon’ involving 20 students, postdocs, and early-career researchers

As the demands of academic research continue to grow, innovative solutions are needed to support scholars in efficiently communicating their findings.

AI tools, when used within a human-supervised process, can help researchers save time by supporting them to focus more on the research itself than on the time-consuming process of writing (see more in a recent Nature survey on AI tools, here).

However, there has been limited exploration of the realities, for researchers, of using AI in controlled, real-life environments.

Last year, we tested our assumptions in book publishing during a hack day, where several researchers were tasked with creating an entire academic book with GPT. They succeeded, demonstrating the potential for AI to save researchers’ time. This year, we asked ourselves, could the same approach also work for high quality publishable articles? Partnering with the School of Social Sciences at the University of Mannheim, we aimed to explore the use of generative AI in creating publishable research articles in the social sciences.

We set out to see how researchers wanted to use AI tools and to assess the effect of large language models (LLMs) on writing productivity and the quality and consistency of scientific output. And our goal was to determine whether AI-supported articles could meet the quality standards required for submission to reputable journals or even do better.

Gathering more than 20 students, postdocs, and early-career researchers from around the world for a three-day hackathon in Mannheim was an exciting and electrifying experience. From the University of Mannheim we had Professor Thomas Gschwend, Chair of Political Science, Quantitative Methods in the social sciences and Professor Marc Ratkovic, Chair of Social Data Science. Each researcher brought with them either their current research or their entire dissertation project.

The hackathon was then organised as a field experiment: on the first two days the attendees in a treatment group were able to use AI tools, while those in the control group were not allowed to use any AI tools at all. The third day was for feedback.

The hackathon was grounded in a very simple reality: we already know that students are using AI tools to support their academic work.

As Professor Gschwend has observed of his students, “these tools excel in organising ideas and refining drafts and that they can support technical tasks like generating code or structuring research manuscripts.” Professor Marc Ratkovic echoed these thoughts summing up that AI has a role to play in research, but we should “consider them as collaborators that can help streamline the writing process, not substitutes for human creativity or intellectual effort.”  What the hackathon helps us to do is to explore where the strengths and limitations of AI are in how researchers want to and are practically using the tools.

The hackathon resulted in several papers being written and a formal experiment write up, all of which are currently at the submission stage. You can keep up to date with the developments around this project by keeping an eye on the collection page here – University of Mannheim exploratory workshop hackathon

Experimenting with different tools, ethically, to help address challenges faced by researchers enables us to better understand the role we as publishers need to play in continuing to support authors with the development of and access to knowledge, whilst ensuring the quality and trust in science remains.

What we do know, as far as new technologies and AI is concerned, is that when leveraged ethically and responsible, it can open a world of opportunities for research. As the aforementioned Nature survey highlighted, over 58% of researchers said that AI had sped up computations that were not previously feasible, and 55% mentioned that it is saving them time and money.

Whether AI tools are fully ready to help researchers save time across all research formats, and globally, is something that as a sector we are still exploring, carefully and thoroughly. However, what this hackathon has reaffirmed is that:

  • AI-tools can help improve work and share results faster – aiding researcher’s writing, coherence, clarity and speed without compromising originality when writing up their data, methods, conclusions and analysis.
  • AI-tools can support research equity by helping non-native English-speaking researchers improve their writing skills and potentially open the door to a global readership.
  • AI-tools need a human in the loop.  Whilst they are providing a lot of opportunities for researchers, as the feedback from the day shows, human oversight is critical to the quality of the piece, helping to review and test the framing, sense and overall integrity of the work.

Niki Scaplehorn is Director Content Innovation, Springer Nature

Do you want to read more content like this? SUBSCRIBE to the Research Information Newsline!

Back to top