The way we navigate data may be changing

Opinion has always been divided about whether it is better to find information in a database by entering a keyword or by working through a series of menus. Which approach is favoured tends to depend on a combination of personal preference, past experiences and training.

Most trained librarians and information professionals are likely to opt for a structured searching model, which is a multi-step search process, usually following the search procedure set by a specific database or service, and typically using a controlled vocabulary. Most end-users, on the other hand, have been brought up on general web search engines and are happier with natural language searching. This enables them to search online using language, phrases, sentences and questions that correspond to the way individuals would normally communicate verbally.

These preferences were confirmed by research conducted by IRN Research in summer 2004. In that survey, 60 per cent of 52 librarians and information professionals stated that their preferred form of searching was a structured search, while only 15 per cent opted for natural language searching. The remaining 25 per cent offered some other search model as their most favoured, with the most popular being an assisted search using a thesaurus. The overwhelming choice for least preferred searching model was natural language searching (67 per cent of respondents). As for their end-users, 71 per cent of librarians noted that they would normally choose natural language searching and only 15 per cent suggested that users would select a structured approach.

Natural language is gaining acceptance
However, when, in 2005, Ovid Technologies commissioned IRN Research to undertake research in the UK healthcare and academic sectors, the topic of search methods was revisited. IRN discussed preferred searching methods for STM databases and services with 35 information specialists, and also asked for their views on the searching preferences of their users.

While results from the 2004 and 2005 surveys are not directly comparable, some general trends can be identified. In particular, the latest research points to greater acceptance amongst information professionals that natural language searching has a part to play in the search process.

According to the latest research, 60 per cent of librarians and information specialists still opt for some form of structured search or Boolean search as their preferred search model but a relatively high percentage - 51 per cent - also accept that natural language searching could play a role in many searches. And, this time, almost a quarter of those interviewed - 23 per cent - highlighted natural language searching as their preferred searching choice.

Again, the end-user preference was overwhelmingly for natural language searching, with 86 per cent stating that the majority of their end-users would opt for this approach. However, many interviewees stressed the diversity of their user groups; while the majority are most comfortable with searches based around natural language, there are some who would be willing to spend more time developing structured searches. But, even here, many users only turn to more structured searches after the first step of natural language searching has apparently failed. Overall, only 11 per cent of all those interviewed would expect the majority of their end-users to try a structured search as their first choice.

Lack of awareness restricts user searches
Survey results suggest that end-users still rely heavily on general search engines for many searches. A limited number have moved onto searching of specialist databases but time pressures, limited searching expertise, and a lack of awareness of the services available are holding back the widespread use of specialist services. These time pressures and irregular searching habits are likely to consolidate the roles of keyword and natural language searching amongst end-users but can they be encouraged to move towards more specialist offerings?

The role of the librarian and information specialist is likely to be crucial in this respect, and there are encouraging signs from specific feedback from the survey. Firstly, more librarians and information professionals are now accepting that natural language searching can be used as an additional search model alongside structured searchings. One cited advantage of this is that the natural language approach can often bring up unpredictable search results. Natural language searching is also identified as particularly appropriate for searches in new areas of research where user-familiarity with exact terms and concepts used might be low.

Another trend is that new medical terms and names are appearing regularly. Only by searching using natural language can the latest terminologies be applied. An indexed database, where a controlled vocabulary or a thesaurus is used for searching, is quickly out of date - and updating puts heavy pressure on resources. However, the availability of alternative keywords, terms, and phrases in a search model is important in enhancing the efficiency of natural language searching. In healthcare, there is often an emphasis on North American terminology. Without alternatives, a natural language search based on European terminology could fall down.

Training is important
In most institutions surveyed, training of end-users on searching techniques is ad-hoc and limited. A majority of interviewees, particularly those in healthcare, admitted that there is little time or insufficient resources to undertake systematic and regular training. With limited training and only sporadic searching, inexperienced users stay inexperienced and these searchers are more likely to continue to rely on general search engines and basic keyword searching. Some suggestions offered to improve training include:

• More on-screen advice and guidance for users on system interfaces and search screens, particularly for services with natural language searching;
• More search guidance and tips, plus online help desks, should be included on intranets and other networks along with opportunities to share searching experiences;
• 'Champions' of natural language searching amongst staff members should be identified and encouraged to share their experiences; and
• Vendors and database providers should offer more training opportunities and material to support in-house training.

Weaknesses of natural language searching
The perceived weaknesses of natural language searching are:

• Just under half of those surveyed have yet to be convinced that natural language searching can avoid the pitfall of too many irrelevant results being retrieved. For STM searches, information specialists felt that end-users often took too little care in filtering out unnecessary and confusing words in the search statement.
• The way searches are ranked according to relevance is usually based on a proprietary system operating behind the service interface. Some information specialists are concerned about the claims of some of these systems and the preciseness and accuracy of the specific methods employed.
• The natural language processing systems in most search engines and online services are models that cannot recognise the context of a question. A key concern for information specialists is that the true meaning of the search will be lost.

The road ahead
If the wider use of specialist information sources is to be encouraged, along with the integration of content into the workflows of end-users, then information services must address the needs and preferences of end-users while, at the same time, striving to educate these users in improved searching methods and more relevant content sources. Improved natural language searching is likely to be an attractive and understandable option for most end-users. Increasing awareness of the possibilities for natural language searching of specialist services offering this model is crucial.

Vendors and content suppliers will have to continue to invest in improved natural language processing models, increase the transparency of these models to users, and offer more training and support in natural language searching.

Further reading
Natural Language Searching - Can Searching the Way We Think Offer a Better Search Solution and are Users Ready to Search the Natural Way?
Ovid Technologies White Paper, March 2005. Further details from: Rossella Proscia, director, International Marketing & Customer Development, Ovid Technologies.

David Mort is a director of IRN Research, a UK-based market research and information company specialising in the analysis of European information and content markets. He can be contacted at dmort@irn-research.com.