Project Cupcake on the rise
Group's intention is to research, discuss and advise on how journal publishers' activities can be measured, writes Phill Jones
The Researcher to Reader conference (https://r2rconf.com/) is an annual meeting of publishers, librarians and academics that takes place every February in London. This year, during a workshop on metrics, an idea emerged to create new types of scholarly journal indicators. The idea generated excitement so we gave it a silly codename, Project Cupcake, and formed a working group.
Traditional metrics focus on the quality and impact of the work that the authors publish in the journal. Cupcake seeks to augment that by measuring the quality of the work that the journal does for the author. As publishing is inexorably changing from a content industry to an information services industry, we believe we should be measuring the quality of those services.
Project Cupcake was started by Tasha Mellins-Cohen of the Microbiology Society, Mike Taylor, and myself, Phill Jones. Mike and I both happen to work at Digital Science but it’s worth noting that Cupcake is not a Digital Science project, it’s very much a community effort. While we’re on the subject of disclosures, I’m also on the advisory board of the Researcher to Reader conference.
If you’ve ever attended Researcher to Reader you’ll be familiar with the unique workshop program that we run there. The workshops are split into multiple sessions over the two days of the event, with the goal of defining and hopefully solving a problem that affects at least one stakeholder in the scholarly communication lifecycle.
In the metrics workshop we began with a fairly open goal of exploring possible new metrics. We asked what governments, funders, institutions, libraries and researchers all wanted to know about research and how it’s communicated – and, equally importantly, what data might be available somewhere to answer those questions.
It quickly became apparent there are already plenty of ways to measure the quality or impact of research itself. Most of these measures are either connected to citations and usage of content or broader, non-academic societal impact such as commercialisation or mentions in policy documents. Essentially, everything that we have already is based around the qualities of the research that a given journal publishes. In other words, the qualities of the work that the journal’s customers do.
Admittedly, when you ask a researcher – particularly a junior one – about how they choose a journal to publish in, they’ll probably talk about the Impact Factor because that is so often the metric they are judged by. If they’re further along in their career, they’ll likely have a more nuanced view and talk about reaching the right readership or the reputation of the editor-in-chief. After that, they’ll complain about things that are important to them personally; overly long review cycles, anachronistic editorial processes that introduce errors and lack of clarity, poor typesetting, and frustrating, burdensome workflows.
On the other hand, institutions, funders and other groups with strategic policy concerns often have questions about infrastructure and metadata compliance that are important to conduct meaningful scientometric analyses. So why, we asked, do we restrict our evaluations of journals to how good they are at identifying articles that will get a lot of citations when they’re responsible for so many more aspects of scholarly communication? Why are there no mechanisms to identify and measure how well publishers are doing at most of the work that they’re paid to do?
So that’s where Project Cupcake comes in. The intention is to research, discuss and advise on ways that more of what journal publishers do can be measured. The idea has proved enormously popular and we’ve amassed a veritable army of talented and eager volunteers to help us sort through the large number of questions that this idea raises. At last count some 50 people have stepped up to help. We’ve sorted the volunteers’ collective ideas into a series of working groups, which will address questions like; how accurately and promptly do publishers supply metadata to Crossref? How well do publishers communicate the types of licences that they use? Is peer-review actually taking place and does the journal have a defined editorial structure? How long does it take to publish in a given journal?
We’ll also be looking at what the eventual successor to Cupcake will look like. Our merry band of volunteers isn’t going to be building technologies to answer these questions: our job is simply to define, inform, and advise the industry on these issues. So in due course we’ll be generating advice on how such a project might be governed and funded.
In September, Tasha Mellins-Cohen will give an industry update on the project at the ALPSP conference in Noordwijk, Netherlands. In the meantime, some more information can be found in the Scholarly Kitchen post that I wrote about the project here: https://scholarlykitchen.sspnet.org/2017/05/22/project-cupcake-designing.... If you’re interested in the Researcher to Reader conference, the call for papers is out now: https://r2rconf.com/call-for-papers/.
