Okapi at City An evaluation facility for interactive IR Stephen Walker and Micheline Hancock-Beaulieu with contributions by Ay§e Goker Lee McCluskey Scarlett Palmer Centre for Interactive Systems Research City University London August 1991 British Library Research Report 6056 © British Library Board 1992 Contents 1 Introduction 1.1 Introduction 1.2 Background 1.3 Development of Okapi at City University 1.3.1 The PCL Okapi system 1.3.2 Parameterization 1.3.3 Updating procedures 1.3.4 Databases 1.3.5 Logging 1.4 Installation and use of Okapi at City 1.4.1 Installation 1.4.2 Use Q u e r y expansion in a library catalogue 2.1 Introduction 2.1.1 Automatic query expansion 2.1.2 Installation of catalogue system 2.2 Background 2.2.1 Library database 2.2.2 Library setting 2.2.3 Network access 2.3 Evaluation methodology 2.3.1 Transaction logs 2.3.2 Search replays 2.3.3 Pre-search and post-search questionnaires 2.3.4 Post-search interviews 2.3.5 Structured interviews of network catalogue users . . . 2.4 Findings 2.4.1 Overall usage of query expansion facility 2.4.2 Effectiveness of query expansion 2.4.3 Perceived usefulness of query expansion facility . . . . 2.4.4 Search intentions 2.4.5 Replay of searches with "MORE" option not used . . 2.4.6 Previous experience with online catalogues 2.4.7 Ease of use 2.4.8 Help facility I 6 6 6 7 7 8 9 9 10 10 10 11 13 13 13 14 14 14 15 15 15 15 16 16 17 17 18 18 18 19 20 20 21 21 22 2 2 2.4.9 User satisfaction with search outcome 2.4.10 Browsing references 3 CONTENTS 22 23 24 24 24 25 25 27 27 28 28 28 29 30 31 32 32 33 34 34 36 36 37 39 39 39 40 41 41 41 42 42 42 44 44 45 46 46 47 49 49 49 51 Transaction log analysis 3.1 Introduction 3.1.1 Source of the data 3.1.2 Sessions and searches 3.1.3 Transaction logs 3.2 Sessions and searches 3.2.1 Aborted searches 3.2.2 Failed searches 3.3 Records retrieved, displayed and chosen 3.3.1 Numbers of records retrieved 3.3.2 Effect of number of records reported on record display 3.3.3 Full records 3.3.4 Relevance judgements 3.4 Query expansion 3.4.1 Takeup of query expansion option 3.4.2 Results of query expansion searches 3.4.3 Source of records displayed and chosen 3.4.4 How useful was query expansion? 3.5 System use patterns 3.6 Differences between classes of users 3.7 Best match searching F r e q u e n t users 4.1 Introduction 4.2 Obtaining the data 4.2.1 Background information about users 4.2.2 Individuals' use of language 4.2.3 Search language in general 4.3 Analysing the data 4.4 Results 4.4.1 General points about frequent users 4.4.2 Suggestions for system modifications Towards an adaptive I R system 5.1 Introduction 5.2 The probabilistic model for an IRS 5.3 The learning component 5.3.1 Inputs to the learner 5.3.2 Formulating and evolving the context 5.4 Testing context formulation 5.4.1 The Okapi system 5.4.2 An example 5.5 Conclusions and future work 4 5 CONTENTS 6 Conclusions 6.1 Automatic query expansion 6.1.1 Query expansion in the library system 6.1.2 Query expansion in searching INSPEC 6.1.3 Heuristics for query expansion 6.1.4 Query expansion: conclusions 6.2 Towards adaptive IR systems 3 52 52 53 54 55 55 57 59 71 77 77 77 77 78 78 79 80 A Okapi system description B Okapi transaction logs C Questionnaires C.l Questionnaires for library users C.l.l Exhaustiveness of search C.l.2 Previous experience C.l.3 Post-search questionnaire C.l.4 Structured interview C.2 Questionnaire for network users D Notes on a frequent user List of Figures 3.1 Transaction log summary 26 59 60 60 61 61 61 62 62 63 63 64 65 65 66 67 67 68 68 69 69 70 70 A.l Welcome screen A.2 Information window for welcome screen A.3 Information offered after user identification A.4 Search input screen A.5 Information offered from search input screen A.6 Search results screen A.7 Replacement of misspelt word A.8 Brief record display A.9 Options from brief display A. 10 Full record display A . l l Brief record display with "MORE" option A.12 Results of query expansion A. 13 Brief display from query expansion search A.14 Record from query expansion A.15 A. 16 Records from second iteration of query expansion A.17 A. 18 Options after query expansion A.19 Choosing the PRINT option A.20 Editing a search A.21 Full record display: INSPEC A.22 Brief record display: INSPEC 4 List of Tables 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 Availability and usage of query expansion Usage of the u MORE" option and search sessions Items selected with and without query expansion Items displayed and selected before and after query expansion Items selected per search after query expansion Search intentions Query expansion in replayed searches Users'experience of online catalogues Ease of use of Okapi Ease of use of Okapi compared to CLSI User satisfaction Perceived number of references looked at Sessions and searches Distribution of number of records retrieved Percent of non-failed searches in which no records were displayed Action following search with no records displayed (CAT1 set) Full records displayed and chosen: searches in which some brief records were displayed Full records displayed and chosen: searches in which some full records were displayed Takeup of query expansion by search result Takeup of query expansion by number of records chosen . . . Records displayed and chosen from query expansion Sources of displayed and chosen records: CAT1 Sources of displayed and chosen records: INSPEC System use Jan-Apr 1991: all identified users System use Jan-Apr 1991: people who had used system before Jan 1 1991 18 18 19 19 19 20 21 21 21 22 22 23 27 28 29 30 31 31 33 33 34 35 36 37 37 50 5.1 o