Chapter 1 Introduction
OPAC research programme
This document presents the findings of a project 'Evaluation of online library catalogues' undertaken by The Centre for Interaction Systems Research in the Department of Information Science at City University. The work was funded by the British Library Research & Development Department as part of an extensive programme of research projects on OPACs which has been carried out by a number of groups in different institutions in the U.K. for more than five years.

Research context
The proliferation of online catalogues since the mid 1980's in both academic and public libraries has led to some concern about their usage. The main characteristic of OPACs is that they are being used by end-users directly without any intermediary / professional help. The implications of this entirely new form of catalogue for the library profession have been and are continuing to be discussed at length. These new library systems need to be monitored and evaluated in a public service perspective, so that librarians can make the appropriate decisions about them. This means assessing not only how OPACs are being accepted and their initial impact on library users, but also how they will affect the information searching habits of a great number and variety of users in the long term. OPACs are providing librarians with the opportunity for the first time to look in great detail at what actually happens when users try to find some material in libraries, especially when undertaking subject searches. Unlike remote online systems, local OPACs offer the possibility of developing their design both in terms of improving the user interaction and providing more sophisticated information retrieval mechanism. The library context in which OPACs are currently used also gives us the possibility of following the information searching process from beginning to end, and not just to observe what happens at the catalogue. The whole search process or interaction of users with the information system can be examined: the recognition of an information 'need', the formulation of a search topic, the search strategies used, the search reformulation, and the document selection at the shelves. 1

Research objectives
A variety of different methods has been used in the past to investigate or evaluate OPACs. To some extent the different methods have reflected the different research objectives and research questions but for the most part these have been primarily concerned with system performance. Our concern here is to focus on the user and the insight which could be gained on user information seeking behaviour through transaction log analysis as well as other methods of enquiry and observation. User performance is thus seen as the means of evaluating catalogue performance. Moreover a holistic approach to the information retrieval task is adopted whereby the entire search process is taken into account including activities which lead to or follow the catalogue consultation. T h e objectives of the project were as follows: 1) To develop an enhanced transaction log facility on a front end as a diagnostic tool for qualitative analysis of searching behaviour at the online catalogue. 2) To assess the effectiveness of transaction log analysis and other methods of eliciting information from the user.

The report
A critical review of the different approaches to evaluation for library catalogues is first presented in Chapter 2. It highlights the methodological shortcomings of the quantitative surveys of traditional library catalogue use studies. The more experimental and user orientated approach of online catalogue research is discussed focusing on the different non-verbal and verbal data gathering methods and transaction log analysis and protocol analysis in particular. In Chapter 3 the methodology of the project is set out. The aim was to improve the effectiveness of logs as a method of evaluation. In the first phase a basic screen logging facility was developed and tested in a library operational setting as a front end to the library catalogue. A holistic approach and combined methodology for data collection was adopted using the logging facility with observation and talk aloud technique. The field trials led to the specification of the enhanced logging facility (Olive) and the formulation of test research questions. These were then used for the set of evaluative experiments which were conducted in the second phase of the project. The conception, specification and implementation of Olive, an enhanced logging facility, is described in Chapter 4. It was implemented as a front end on a P C to allow for searches on the library catalogue to be intercepted and online questions to be put to the searcher under certain conditions. Chapters 5 reports on the results of the preliminary experiment and subsequent evaluative experiments which addressed test research questions pertaining to subject searching strategies, search formulations and search outcomes and user satisfaction. The experiments explored and validated the different ways of eliciting information 2

from users including pre-search / post-search, offline / online and interactive online questionnaires, as well as transactions logs and the replay of searches. A discussion of the results and critical assessment of the different evaluative methods follows in Chapter 6 with conclusions and recommendations for future work in Chapter 7.

Project staff
The two year project started in October 1987 under the direction of Professor S. E. Robertson. Other project staff included, Dr. M. Hancock-Beaulieu, Research Officer and Mr. C. Neilson, Research Assistant. Programming support was provided under contract by Dr. C. Pope and Mr. T. Wright of the Systems Science Department, City University.

Acknowledgments
In the course of the research the project staff were in contact with an informal advisory panel of researchers in the field of OPACs. Their contribution at different stages and in particular their comments on the interim report were appreciated. They included Christine Borgman, Philip Bryant, Charles Hildreth, Michael Lesk, Karen Markey and Stephen Walker. We are also grateful for the assistance and support we received from a number of other people: Efthimis Efthimiadis who produced the bibliography on OPACs, Mr. R. V. Fox and Peter Butcher of City University Library and Karen Merry and Derek Greenwood, our BLRDD project officers. Last but not least we thank the hundreds of City University students who agreed to participate in our experiments by searching the library catalogue.

3