FEATURE

Assessing online usage

Joachim Schöpfel and Chérifa Boukacem-Zeghmouri report back from a recent symposium on assessing online usage of academic resources held in Lille, France

Last November Lille, in the heart of Europe, hosted the first international conference on usage-related approaches to academic digital libraries.

The first message of the Lille conference was unambiguous: usage assessment is a hot topic for the whole value chain of the scientific information market. This market is fundamentally international, and so is research on usage. Half of the speakers, sponsors and members of the programme committee came from countries other than France. However, scientists and professionals from France, the UK, Germany or Japan ask the same questions, use the same tools and methods and are confronted with the same kinds of difficulties.

Linked to the rapid globalisation of the industry this is a significant change that allows for international cooperation in usage assessment and global understanding of scientific information behaviour. Keynote speaker David Nicholas from the UCL CIBER team in the UK described the web generation as ‘being frenetic, bouncing, navigating, checking and viewing, as well as promiscuous, diverse and volatile.’ They are net citizens of the global village, despite their different origins and locations. ‘The web is having a profound impact on how we conceptualise, seek, evaluate and use information. What Marshall McLuhan called “the Gutenberg galaxy” is imploding, and we don’t know if what will replace it will be better or worse,’ he said.

David Nicholas also confirmed the importance of deep log analysis methodology for usage assessment. Based on seven years of data and millions of digital footprints in e-books, e-journals, e-learning and e-cultural databases covering every subject and coming from every country, the CIBER team has gathered a formidable evidence base to understand what people did, not what they say they did, or wish they did. Log analysis allows for separate evaluation of activity metrics (such as number of full-text downloads and pages viewed), information-seeking characteristics (type of content viewed and search style) and user characteristics (geographical location or organisation, for example).

The French research combines log analysis, evaluation of usage statistics, survey data (such as interviews with scientists from different disciplines and organisations) and direct observations. Qualitative research allows for better understanding of the quantitative usage data in terms of behaviour, intentions and information strategies, for example.

Same data, different use

The third message of the Lille conference is that one can use the same usage data for at least four goals. The first is scientometrics, where usage data contributes to the assessment of scientific performance and production of scientists and institutions. The second use is in services. Publishers, vendors and libraries use access data to improve service quality and added value. Thirdly, there is return on investment (ROI). Institutions need this data to evaluate the return on investment in online resources acquisition and digital library tools. Last, but not least, usage data can help to understand consumers’ behaviour and link this behaviour to the specific institutional or community environment. For instance, this kind of analysis helps shape new training programmes for future library and information science professionals.

However, two issues call for attention. There is the risk of uncoupling between academic library services and skills, and their traditional patrons muted into net citizens. There is also the association between usage behaviour and the technical, commercial and legal characteristics of the online resources. Scientific research on usage must not neglect the evaluation of the online services on offer, such as portals, digital libraries and archives. The missing link is needed between ergonomics and usability criteria and usage data.

Open archives

Discussions at the conference focused on usage assessment of institutional and other open repositories. After some years of existence, the open-access movement is now in the same situation as digital libraries were at the beginning of the big deals. Usage data are often incomplete and not compliant with other repositories or online resources. Also, there are specific and unresolved problems, such as missing unique identifiers, multiple deposits, and a mixture of full text and records. The Lille conference discussed German, French and UK projects that are developing a framework for open-archive usage data. Much work remains to be done. Nevertheless, the evident synergy between the different initiatives will facilitate the emergence of an international standard such as COUNTER.

There is a need for standardisation of usage data, terminology, metrics, procedures such as log analysis and related services. We have to know what exactly we measure and how we do it. The COUNTER project remains the central model for usage assessment, even if new developments are required, such as usage reports at the individual item level and log analyses. Projects like PIRUS (JISC) and OA-Statistics (DINI) will continue to contribute to the standardisation and interoperability of metrics and tools.