X m p r o v ± n g ^3 L_J tz> j j e? cc " t r^epizr^iL^v^^L i n o n l i n e cc a a ^ aa L CD eg L_J € ? ifa E 2. Relevance feedback and query expansion 5 tephen and Rachel WaLker De Vere British Library Research Paper 72 1383 British Library Research and Development Department R r ep f aa cz ee * This report is about improving subject access in computerized bibliographic information retrieval systems which are intended for end users rather than for information specialists. It contains a description of some retrieval systems which were used to provide subject access to a Library catalogue database, and of some experiments which were done using these systems. It reports the most recent of a series of projects the Okapi projects - which began in 1982 and have been funded by the British Library Research and Development Department and hosted by the Polytechnic of Central London. Gill Venner Gill Venner, one of the team of three which designed, developed and reported the original Okapi system during 19B3-1S85, died while this report was being written. The report is dedicated to her memory. Rcknowledgment s This work would not na^e been possible without the help and support of the British Library Research and Development Department. Particular thanks are due to Karen Merry and Derek Greenwood, our present and former project officers. We are also mast grateful to Neil McLean CProject Head and formerly Director of Information Resources Services, Polytechnic of Central London], and to the staff of the Polytechnic's Computing Services. People who gave valuable advice on the design of the retrieval systems and/or the experiments include Win.' ec Rbbott, Maura Coghlan, Ellen Gredley. Charles Hildreth, Mark Lansdale, Martin Porter, Stephen Robertson and Peter WiLlett. Efthimis Efthimiadis, Michel.me Hancock-Beaulieu, Colin Neilscn and Stephen Robertson read portions of the manuscript and made valuable comments and suggestions. Credits It would not have been possible to design and program the experimental retrieval systems used in this project without the foundation provided by previous Okapi systems. Major contributors include Nicky Johns, Richard Jones, Nathalie Mitev, Julie Porteous, Gill Venner and Stephen Walker. The evaluation experiments for this project were designed and carried out by Rachel De Vere as part of the work for her MSc in Information Technology at Loughborough University. Parts of Chapters 3, 4 and 5 of this report are adapted from De Vere's MSc dissertation. Stephen Walker City University Rachel De Vere University of Sheffield -in- CCDNTENTS 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 2 2.1 2.2 2.3 INTRODUCTION The Okapi projects Notes on terminology Subject access in online catalogues Subject access in Okapi '86 Query expansion Feasibility studies Development of the experimental system Okapi '88 QUERY MODIFICRTION THROUGH RELEVANCE FEEDBRCK Retrieval techniques Term weighting and relevance feedback Query modification 2.3.1 Sources of terms for query modification 2.3.2 Weighting and term selection Query modification in end user retrieval systems 2.4.1 Term selection by the user 2.4.2 The CITE catalogue at the NLM 2.4.3 The online catalogue at the Scott Polar Research Institute 2.4.4 The INSTRUCT system at the University of Sheffield SYSTEM DESIGN a DESCRIPTION 1 1 1 2 5 5 6 7 9 9 11 13 13 14 14 14 15 16 17 13 13 19 20 20 20 20 21 21 22 23 24 24 25 26 27 27 28 29 29 30 32 2.4 3 3.1 3.2 3.3 3.4 3.5 Introduction The bibliographic database Hardware and software Dialogue style Obtaining and processing user input 3.5.1 Obtaining input 3.5.2 Input preprocessing 3.5.3 Parsing and lookup 3.6 Search processing 3.5.1 Term weighting and the merge 3.6.2 Initial search results and options 3.7 Record displays 3.7.1 Brief record display 3.7.2 Full record display 3.8 Obtaining relevance judgments 3.3 Query expansion 3.9.1 Term extraction and weighting 3.9.2 Selection of terms for query expansion 3.9.3 Query expansion search - screen display and options 3.10 Shelf-order browsing 3.11 Other options Screen illustrations EVRLURTION 1 Introduction 2 Formal, experiments 4.2.1 Rn example 4.2.2 Formal experiments with interactive systems 3 Experiments with real users 4.3.1 The NLM online catalogue comparison 4 Evaluation techniques for highly interactive systems 5 Rims of the evaluation B Planning the evaluation 7 Performance measures 4.7.1 Recall, precision and efficiency 4.7.2 Relevance, recall and precision in the present experiment 4.7.3 Transaction Log analysis 4.7.4 Users' opinions B Method 4.8.1 4.8.2 4.8.3 Subjects Tasks Experimental procedure 3 Objective relevance assessments RNRLYSIS & RE5ULT5 1 Introduction 5.1.1 Terminology 5.1.2 Source and processing of data General comparison of the systems 5.2.1 Statistics on efficiency and effectiveness 5.2.2 User opinions on system ease and helpfulness 5.2.3 User opinions on system usefulness Use and performance of the query expansion and classificat browsing facilities Query expansion in detail 5.4.1 Statistics 5.4.2 QuaLity of Lists of records from query expansion 5.4.3 Users' comments on query expansion Classification browsing in detail 5.5.1 Statistics 5.5.2 QuaLity of Lists of records from classification browsing 5.5.3 Users' comments on classification browsing "Objective" precision of chosen records User comments about the systems 5.7.1 Choice of search terms and retrieval of non-relevan records 5.7.2 Record content and display; the recognition of relevance 5.7.3 Problems connected with the list of chosen records 5.7.4 General interaction and presentation How people searched the systems 2 3 4 5 6 7 9 -vi- 6 CONCLUSIONS & RECOMMENDATIONS 73 73 73 74 75 76 71 77 78 73 80 80 80 81 81 82 6.1 6.2 6.3 6.4 6.5 Performance of the experimental search systems Query expansion 6.2.1 Towards heuristics for offering query expansion 6.2.2 The collection of relevance information 6.2.3 Computational implications of query expansion Classification browsing 5.3.1 Why classification browsing was relatively unsuccessful 6.3.2 Classification browsing in a live system 6.3.3 R note on synthetic Dewey numbers Further work 6.4.1 When and why does automatic query expansion work? 5.4.2 Rutomatic query expansion with non-catalogue databases 6.4.3 Catalogue systems with enhanced records 6.4.4 Towards highly interactive systems: user monitoring as a tool for guiding interaction Concluding remarks RPPENDIXES 1 2 3 4 5 6 7 8 File structures Input processing Questionnaires Task sheets Annotated extracts from a log file Assessors' instructions Class browsing results Example searches 1 Extracts from a live search en the query expansion system 2 Notes en a search of the "full" system 83 85 36 31 35 35 101 104 104 107 '10 REFERENCES -vii-