Measuring the impact of OA content

Share this on social media:

Tim Lloyd

Measuring the impact of open access (OA) agreements is critical for developing sustainable open access business models, writes Tim Lloyd. And for an increasing number of funders, an important ingredient of impact is the audience. Yet existing analytics don’t do a good job of understanding how communities are engaging with OA content.  

With OA content now accounting for the majority of journal articles published1, we need a better understanding of which stakeholders across our industry rely on OA analytics, and how they want to use them.

To this end, LibLynx and PLOS partnered last year to develop the next generation of OA analytics. The first step was to engage with PLOS stakeholders - institutions, funders, and consortia - to understand how they wanted to assess the value of OA publishing. 

The limitations of traditional approaches

Traditional quantitative metrics, such as citation analysis or the number of downloads, provide easily comparable numbers but little depth - was the usage from the communities targeted, or a bunch of bots and pirates vacuuming up free content? Altmetrics add a valuable sense of the attention that articles receive, but underplay value for communities that don’t engage with media in the same way.

COUNTER Reports seem an obvious answer to this problem, and one that libraries are already very familiar with, but they are engineered for a very specific use case - helping librarians understand and compare usage of paid content in order to make informed decisions on acquiring content to meet their library’s learning and research goals. This traditional COUNTER use case rightly focuses on the aggregate numbers because the audience is already defined as the subscribing institution. But there is no standard for how to attribute usage more broadly, or to offer alternative methods of analysing that audience.

However, this does not mean that COUNTER has no role to play.  Far from it, because 'metrics are the gold nugget at the heart of COUNTER'2. Think of COUNTER metrics as a defined set of lego bricks that we can re-combine to create new, valuable analytics reporting.

Who needs OA analytics?

Back to our partnership with PLOS. Our research over the last three to four months has identified a wide range of industry stakeholders:

  • Research institutions that are typically generating the research that is published. Most obviously, we’ve got the traditional library role, which sits at the core of the COUNTER Reporting use case. Librarians are already tracking usage of OA content as part of licensed collections, and those that also play a role in pure OA publishing will be equally interested in monitoring that usage. There are also institutional roles that sit outside the library and are focused on research management, such as the senior research officer3.  These roles are more interested in understanding how usage of OA content ties into the institutional research priorities.

  • Publishers that publish OA content. These can be the same research institutions or dedicated publishing organisations like PLOS. There are development roles that need to understand which organisations are getting value from OA content in order to identify potential future sources of funding, as well as editorial roles that want to understand the subjects and topics that are engaging the community.

  • Authors of that research. Authors want to understand the impact of the research they publish. In some cases, this information can influence their choice of publisher.

  • Funders that pay for the research to be published. These can be institutional budgets or separate entities, such as Wellcome or the Gates Foundation. Funders also want to understand the impact of the research they funded. Did it reach the communities they were targeting, or perhaps it also got engagement from new communities that they previously weren’t aware of?

  • Various intermediaries perform a variety of functions that support the publishing workflow, such as service providers like KU, consortia like Jisc, and distributors like JSTOR.

  • Last, but not least, the broader community that is interested in reading that research – a group often overlooked, but of key importance to funders vested in delivering benefits outside of narrow research interests, such as publicly-funded institutions.

And what metrics are they interested in?

While we’re still digesting the results of our research, there are some clear themes emerging.

COUNTER metrics are highly valued. While the nature of COUNTER reports will develop over time to incorporate OA content, it’s clear that the value of consistent, credible, and comparable underlying metrics is as important as ever.  Institutions want to understand the value they get from their publishing relationships, regardless of whether the content is subscription or one of the increasing flavours of OA, and across both the content they pay to publish AND the content they consume to support learning and research.

Understanding your audience for research is essential. While data privacy rightly ensures individual anonymity, it’s still possible to quantify usage by geography and (where IP address matching allows) by organisational name and category. This can identify new, valuable communities engaging with research that have previously been ignored.

We need to support more diverse use cases. OA stakeholders want far more granular detail in addition to traditional aggregated reporting; visually-rich layouts that are easy to consume in addition to tabular formats, and real-time reporting in addition to periodic reports.


1. According to data from 51% of journal articles published in 2020 are available through OA.

2.  Jeremy Morse, Director of Publishing Technology, Michigan Publishing.

3. Read Roger Schonfeld’s fascinating December Scholarly Kitchen post to better understand this developing role.

Tim Lloyd is founder and CEO of LibLynx