Evaluation of Online Catalogues: an assessment of methods
Micheline Hancock-Beaulieu Stephen Robertson Colin Neilson

British Library Research Paper 78 1990

British Library Research and Development Department

British Library Cataloguing in Publication Data Hancock-Beaulieu, Micheline Evaluation of online catalogues: an assessment of methods. (British Library research paper, ISSN 0269-9257; 78). 1. Libraries. On-line catalogues I. Title II. Robertson, Stephen III. Neilson, Colin IV. British Library, Research and Development Department V. Series 025.3132 ISBN 0-7123-3226-X

© T h e British Library Board 1990 The opinions expressed in this research paper are those of the authors and not necessarily those of The British Library. RDD/G/808

British Library Research Papers are published by The British Library Board and distributed by The British Library Publications Sales Unit, Boston Spa, Wetherby, West Yorkshire LS23 7BQ, UK. In the USA they are distributed by the American Library Association, 50 East Huron Street, Chicago, Illinois 60611; in Canada by the Canadian Library Association, 200 Elgin Street, Suite 602, Ottawa, Ontario K2P 1L5; in J a p a n they are distributed by Kinokuniya Co Ltd, P O Box 55, Chitose, Tokyo 156; and in India, Burma, Pakistan, Nepal, Bangladesh and Sri Lanka they are distributed by Arnold Publishers (India) Private Ltd, A B / 9 First Floor, Safdarjung Enclave, New Delhi 110029.

Printed in Great Britain by The British Library.

ii

Abstract A project investigating tools, techniques and methods for the evaluation of interactive library catalogues is described. Methods used in other projects are reviewed. The emphasis of the investigation was on diagnostic methods, and on the use of the catalogue in the wider context of user information seeking behaviour. Studies on users of the City University Library catalogue indicated that (a) traditional transaction logging is quite inadequate as a method of evaluation; (b) various enhancements can be introduced to greatly increase the usefulness of transaction logging; (c) a fuller understanding of the processes involved requires study of the user's behaviour away from the catalogue. Useful enhancements to transaction logging include: a) full-screen logging b) a playback facility c) pre- or post-search, online or offline questionnaires d) interactive, in-search questionnaires e) talk-aloud. A front-end system (Olive) has been developed to allow the first four enhancements. The experiments also emphasise the extent and importance of subject-searching or hybrid searching activity, as opposed to specific item searches.

iii

Dr. Micheline Hancock-Beaulieu is Research Co-Ordinator at the Centre for Interactive Systems Research, Department of Information Science, City University. Her research has been concerned with information seeking behaviour, evaluation methodology and information systems for the Humanities. Professor Stephen E. Robertson is Head of the Department of Information Science in City University and Director of the Centre for Interactive Systems Research in the Department. His research interests are in probabilistic models, relevance feedback, and the evaluation of IR systems. Colin Neilson has previously worked on projects on Bibliographic Control of Computer Files and on Classification for Community Information. His research interests include computerisation and subject indication in working with the BLISS C L A S S I F I C A T I O N
ASSOCIATION.

iv

Contents
1 Introduction 2 Approaches to Evaluation for Library Catalogues 3 4 Methodology Olive—an enhanced logging facility 1 5 15 23 35 59 63 65 69 73 79 83 87

5 Results 6 7 Assessment of Evaluative Methods Future directions

References Appendix A Appendix B Appendix C Appendix D Appendix E

v