Usage: the missing measure of impact

Share this on social media:

Issue: 

Tasha Mellins-Cohen

COUNTER reports have an integral role to play in our wider scholarly communication system, writes Tasha Mellins-Cohen

Let’s be honest, many publishers today view COUNTER reports as a library service, something the sales team can use in negotiations, but that has limited value in the wider organisation. For libraries, COUNTER reports help calculate cost per download and reveal which services are ripe for cancellation or renegotiation.

Both views are restricted to usage metrics for subscription content. In our current environment, with the expansion of Open Access and diversification of research outputs, a metric that helps define the value of subscription content risks becoming defunct. 

I argue, however, that COUNTER reports have an integral role to play in our wider scholarly communication system. For all that we debate many matters, one desired shared by libraries, publishers, funders, and researchers is to be able to demonstrate impact not at the level of a title, but for each article, book chapter, or dataset. Historically we have used citations to indicate impact, a lagging indicator that may take years to accrue. In more recent years we’ve adopted alternative metrics to track social media mentions and other such indirect indicators of interest. What has been left on the table is usage: a direct, immediate measure of impact in the form of active engagement with either the metadata or the full text of a work. This is information many publishers already have at their disposal in the form of COUNTER metrics. 

I would like to see COUNTER take its place as an integral part of the wider scholarly communication system, vital to researchers, institutions, funders, and publishers as one of a suite of metrics demonstrating the impact of research, credible to all parties, and responsive to changing community needs. 

Vital

As I have already said, every part of our community is looking for ways to demonstrate the impact of research. We are all well aware of the flaws and biases inherent in using a single metric such as citations for this purpose, and item-level COUNTER metrics have a role as one of a suite of impact metrics. That will only be achievable, however, if we increase understanding of and demand for item-level COUNTER metrics among the community. 

COUNTER volunteers spend a lot of time and effort on outreach, but many of those who come along to webinars, respond to surveys, or otherwise engage with COUNTER are drawn from the same relatively small group of people. By collaborating with industry bodies we hope to reach a wider audience, learn from the leadership of those groups, and hopefully bring some influence to bear in developing conversations about usage and impact metrics. 

That collaboration and reach aids in our community engagement initiatives, not just within the library and publisher communities but among the providers of open scholarship services such as data repositories and, through institutions and learned societies, with the researcher community themselves. Publishers have adapted rapidly to serve the growing and changing needs of researchers, funders, and institutions over the last few years. If these groups start to request article level, audited COUNTER metrics – information that publishers already have – I suspect such badging will be quick to arrive. 

Credible

As things stand today, COUNTER is widespread but poorly understood. Alongside work in developing the value of COUNTER, we need to ensure that understanding of the standard is as widespread as possible. In this area, we should again be collaborating with our industry colleagues to determine what education and so forth would be valuable to their members. We should be learning from areas beyond scholarly communication about how best to provide educational content in an engaging, accessible fashion that will help to build trust in COUNTER metrics. 

And of course, we need to carefully consider audits. In 2021 we undertook a deep dive into the audit process with a series of focus groups, and that information is being used to inform development of the upcoming Release 5.1 of the Code of Practice (which will be released for consultation in mid-2022). 

Responsive

The whole COUNTER volunteer community has made significant efforts over the last several years to seek out, consider, and respond to community needs – for example, our recent consultations with publishers and providers about how we could improve the release and audit processes, as well as consultations with librarians about their use of search metrics and with our Open Access Advisory Board about better accommodating OA within the Code of Practice. Our annual member survey equally aims to allow us to respond to members’ needs. However, the development of any standard takes time and in the interim between COP releases the world moves on. 

In many cases, the existing Code of Practice is flexible enough to accommodate those changes and still deliver value – to take one example, our aggregated usage statistics (“The World”) reports have facilitated Open Access reporting since the original Release 5 went live five years ago. Engaging with our community to communicate that flexibility is essential, but we must also understand where the Code’s flexibility falls short. 

One challenge is that COUNTER releases often require providers and publishers to do development work, which may be seen as a chore that could take a back seat to more obviously interesting projects. Which rather brings us back to my first argument: if we can position usage metrics alongside citations and altmetrics as a measure of impact, would publishers, funders, and researchers start to value usage as much as librarians do? 

Get involved

COUNTER is a community-led organisation – please do get in touch with me if you want to be involved: tasha.mellins-cohen@counterusage.org

Tasha Mellins-Cohen is project director at COUNTER