British Library R & D

Report 5553

R E S E A R C H 0 N R E L E V A N C E W E I G H T I N G 19 7 6 19 7 9

K ^ S p a r c k C. A.

J o n e s

W e b s t e r

Computer Laboratory U n i v e r s i t y of Cambridge Corn Exchange S t r e e t Cambridge CB2 3QG England

1980

i

ACKNOWLEDGEMENT

This work was supported by the British Library Research and Development Department under grant SI/G/210 to Professor M.V. Wilkes of the Computer Laboratory, University of Cambridge. We are grateful to the University Computing Service for permitting use of the University's 370/165 computer. We thank Dr S.E. Robertson in particular, and also Professor C.J. van Rijsbergen, Dr D.J. Harper, and Dr R.M. Needham for many useful discussions. Data used by the project was supplied by Professor C.W. Cleverdon, by UKCIS via Dr D.C. Veal, by Inspec through Mr L. Evans, and by the National Physical Laboratory through Dr P.K.T. Vaswani. We are very much obliged to them all for making their material available.

Karen Sparck Jones July 1980

ii

ABSTRACT

This report describes experiments on relevance weighting of search terms. The tests were designed to study the behaviour of relevance weights in relation to other forms of request modification, and in the context of different collection environments, specifically those defined by variations in the amount of relevance information available for weight calculation. The results show that relevance weighting can be very effective in a wide variety of situations, and effective even in very unfavourable circumstances.

iii

CONTENTS PART J[
:

TEXT : Background

Section 1 2 3 4 5 Section 1 2 3 4 4.1 4.1.1 4.1.2 4.1.3 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.3 4.3.1 4.3.2 4.3.3 4.3.4 5

. A

Object of the work Test data General testing strategy . Performance representation Tabulation of search output B : Experiments

Overview of the experiments The framework for the experiments Relevance weighting The tests Request membership Regular environments Variant environments Conclusions on request membership Request weighting Weighting formulae Prediction Specific formula tests Alternative performance representations Variant collections Qualitative conditions Quantitative conditions Specific condition tests Cross checks and alternative representations Iterative searching

iv
Section 1 1.1 1.2 2 3 4 C Discussion Analysis of the test results Request membership Request weighting Boolean searching Comparable research Conclusion FIGURES, GRAPHS AND TABLES

PART Figures Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Graphs Graphs Graph Graphs Graphs Graphs Graphs Graphs Graphs Graphs Tables

A1 A2 A3 A4 A5 A6 A7 A8 A9 B1 B2

Test data summary Test raw material Test collection derivation Test collections The profile collections Collection comparison Subsidiary collections Term retrieval facts Performance representation methods Special cases for weighting functions Numbers of retrieved documents dv re dv dv dv dv re re dv Request membership
ti ft

1- 2 3 4- 9 10-12 13-14 15-20 21-26 27-29 30-35

Relevance weighting, F4 F1 H1 U1 F4 U1 variant collections

Note on the organisation of the Run Tables Key Table : Specification of Tables, Specification of Runs

V

Summary Table Summary Tables, variant collections v1, v2 Main Tables MT, MW, MR, MS, MSW, MSR, MC, MCW, MCR, ME, MEW, MER Main Tables, variant collections v1/MR, v2/MR, v2/MSR, v2/MCR, v2/MER Secondary Tables SrcT etc., SrcR; variant collections v2/SrcR, v2/SrcCR SdrT etc., SdrR; variant collections v2/SdrR SprT etc., SprR; variant collections v2/SprR Str variant collections v1/Str, v2/Str SrrT etc., SrrR; variant collections, v2/Srr ScrT etc., ScrR variant collections, v2/ScrR, v2/ScrCR Other Table REFERENCES GLOSSARY