159

9

Conclusions and recommendations
Summary of results of the project

9.1

The team set out to produce an online catalogue which would: allow access to the bibliographic files of a small to medium library and operate on a low-priced LAN, and be readily usable without training or experience, without sacrificing effectiveness or being tedious for experienced users. The first goal has been achieved. The prototype OPAC, Okapi, could serve as the principal means of access for small to medium collections (probably up to about half a million bibliographic records, and more with comparatively minor software enhancements). It is difficult to make an objective assessment on the second goal (see Section 9.4.6 for a discussion of some of the problems of the evaluation of OPACs) but the evaluation results given in Chapter 8 suggest that it has been fairly successful: it is undoubtedly easy to use, and appears to be relatively effective. It must be emphasised that the prototype Okapi is not intended to be a finished system. There are a number of ways in which it is simply unfinished: there are, for instance, some access points which have not been made available to users, and the subject search function could readily be improved. The search programs contain mistakes and inconsistencies. There is no provision for the input or editing of records. Apart from this, all OPACs covering loan collections should be linked to circulation control. The importance of Okapi lies in the fact that it has demonstrated that an OPAC can be both easy to use and effective, and that it is in many respects a good test system for further research and development.

160 9.2

9. Conclusions and recommendations OPACs — advantages and disadvantages

The paragraphs below, labelled (A)-(E) and (l)-(3), list some of the advantages and disadvantages of OPACs. The cross-references are to sections later in this chapter in which there is further discussion of some of the ways by which OPACs might be improved, together with some fairly specific recommendations for further research and development. Potentially, OPACs provide the opportunity for libraries and their users to achieve the following objectives: (A) (B) Powerful and flexible subject access (Section 9.4.1). Specific item access which is as good as that in conventional catalogues for precisely specified items, and better for inaccurately specified ones (Sections 9.4.2 and 9.4.3). Access to contents, summaries, or even full text ( Sections 9.4.1 and 9.4.5). Location and availability information. Access to other collections.

(C) (D) (E)

As yet, none of these objectives, with the exception of (D), has been achieved in a live, general purpose OP AC. Further, OPACs (or rather their users) suffer the following disadvantages in comparison with most conventional catalogues: (1) They are often more difficult to use and to learn than conventional catalogues and less tolerant of mistakes and misapprehensions. To use even the most elementary facilities (well-specified known item search) requires a higher degree of involvement (Sections 9.4.2 and 9.4.4). It seems that many users do not regard OPACs as being catalogues in the traditional sense (Chapter 8); presumably it is the interactive nature of OPACs which gives rise to this attitude. Thus an ineffective OPAC may raise users' expectations initially, followed by disillusionment. More seriously, users are too ready to be satisfied with an incorrect or inadequate OPAC response to their queries — the OPAC seems to give an impression of infallibility which traditional catalogues do not have (Sections 9.4.4 and 9.4.6).

(2)

9. Conclusions and recommendations (3)

161

The poor quality of the display and the small amount of information which can be shown at one time leads to browsing facilities which are in some respects inferior to those provided by microform catalogues (Section 9.4.3). LANs and library automation

9.3

Library automation functions differ widely in their importance for library users and for library staff and management, and in the demands they make on software and hardware resources. For users, access to the catalogue is undoubtedly the most important, closely followed by circulation control. In lending libraries the catalogue access and circulation functions must be combined. Clearly, catalogue access needs cataloguing facilities, so the provision of these is also a primary function. Functions of secondary importance include acquisitions, serials control, management information and other office automation procedures. Circulation control makes the heaviest demands on the hardware because it requires instant updating of copy and user files, a guaranteed quick response time and high reliability. The public access catalogue needs far more terminals and good response time, and is unquestionably the most difficult to design and to program, but files need only periodic, batch, updating. Catalogum^ is not a heavy load on resources unless it is required that new records be instantly accessible to users of the public access catalogue. This project has demonstrated that a LAN is an eminently suitable host for the catalogue of a smallish library, and that other activities can take place simultaneously with the use of the catalogue. The hardware and system software used would be suitable for the full automation requirements of most libraries of up to a million or so bibliographic records. For installations towards the top of this range it would be necessary to have more than one file server and disc drive. This gives the further advantage that the system can still work if one disc drive fails (Section 3.5, footnote); this is particularly important for circulation control. Large libraries considering the use of a LAN should study the Geac installation at the Bobst Library of New York University [1], A set of LANs, each with one or more file servers, possibly linked by fast telephone lines, would work well as the host for an OPAC for a multi-site library, but circulation control, which needs real-time updating on all

162

9. Conclusions and recommendations

sites simultaneously, might not be so satisfactory as on a lightly loaded multi-user mini installation. For some multi-site libraries the OPAC might most practically be updated across sites by reloading the catalogue files at each site using a cassette or other tape (at 1000 characters per second on a fast telephone line, an indexed file of a quarter of a million catalogue records would take one or two days to transfer). With present technology and current library software, the difference between the use of a LAN and a multi-user single processor system for library automation is minimal and ill-defined: it is difficult to think of things which can be done with one and not with the other. There may be a cost difference. In the longer term it will be important for the design of highly responsive interactive systems (Section 9.4.4) that each user station should have its own processor(s) and a large amount of local memory (core certainly, backup possibly), and stations have to be linked (e.g. for circulation). This does call for a LAN or other multiprocessor system, and fast, accurate data transmission. 9.4 9.4.1 Some conclusions and recommendations for research and development On the enhancement of subject searching in OPACs

This is too large a topic to be covered exhaustively here; interested readers are recommended also to read the recommendations of Markey [2], of Kaske and Sanders in [3] and of Walker in [4]. There are really two approaches to this problem. The first is to index everything in the (preferably enhanced) MARC record which could possibly have subject content, then to make this indexing more fuzzy by, for example, the application of word stemming rules, and then to do a completely post-coordinate search on what the user enters. This might be called the sledgehammer approach. It is highly likely to result in the retrieval of something relevant, when there is anything, but at the cost of a large proportion of "false drops", particularly in a large file. The second approach is to make the best possible use of the controlled subject information in the records by helping searchers to find and to make use of assigned subject headings and/or classification codes. When this is the only method of subject access in an OPAC, as it is in some of the earlier systems, many searches fail because the user cannot find an initial entry point to the subject indexing. In any case, facilities for browsing

9. Conclusions and recommendations

163

subject headings or class marks must be provided. The burden is placed on the user rather than on the system. In a notable study of the use of an LCSH (card) catalogue in an academic library, Bates [5] found a strikingly low success rate; catalogue familiarity had a significant beneficial effect but, alarmingly, subject knowledge was of no help, being if anything slightly detrimental to users' success in finding an entry. It is fairly obvious that both "free" and "controlled" approaches must be utilised for any collection other than the smallest (Section 2.5).
METHODS OF IMPLEMENTING ENHANCED SUBJECT ACCESS

Hildreth [6] says "users' difficulties with online subject searching are primarily of two kinds: (1) matching the terms which they use to express their topic to the terms indexed in the online catalog; and (2) finding terms related to their search terms, including broader terms and narrower terms". While we do not disagree that these are the problems, observation and logging of catalogue use show that it is not realistic to expect the user to play the active part in these tasks. Many users have neither the persistence nor the skills required. It follows that the OPAC itself must play a larger part, and make the minimum demands upon the user. Hildreth's first point is best satisfied by ensuring that there are as many entries to the catalogue as possible, that is by providing the sledgehammer approach. There has been a considerable amount of research into providing additional access points beyond what is derivable from the content of MARC records. The "BOOKS" project of Atherton and others [7] investigated the utility of adding terms from contents pages and back-ofthe-book indexes, with encouraging results. Although the average extra cataloguing time per item was only about ten minutes, it seems unlikely that the larger cooperatives, or even many individual libraries, will be able or prepared to make even this rather small investment. Currently, Markey and others are doing an elaborate study of the additional entry points (and browsing facilities) which can be provided by making available online extracts from the Dewey schedules and relative index, linked to the MARC records by means of the class number [8,9 — also several interim reports available from OCLC]; this can undoubtedly provide more subject entry terms, and, since the Dewey

164

9. Conclusions and recommendations

files are available in machine-readable form, it does not require any additional cataloguing effort. However, it does appear that the language of the Dewey schedules, being designed to help cataloguers rather than users, is rather far from the language of many subject search queries. On Hildreth's second point, the problem is one of finding related records rather than related terms. The Okapi logs do indeed quite often show revision of queries following an unsatisfactory search (a typical example is ''London Broadcasting Company" followed by the broader query "independent radio"), but it is not possible for most OP AC searchers to learn the kinds of skills which are used by experienced intermediaries in conventional online searching.
RELEVANCE FEEDBACK AND RANKED OUTPUT

Much research in IR has been directed towards investigating ways of automatically finding records which have a high probability of being similar in subject content to those which match the terminology of the query. Such methods include clustering techniques (terms are divided into groups on the basis of their probability of co-occurrence, and the system will retrieve records indexed by as many as possible of the terms which regularly occur in conjunction with index terms matching the query), and the use of relevance feedback: the query is modified by, for example, adding terms from records judged by the user to be relevant to the query, and reprocessing the search. Harper's thesis [10] contains a rather comprehensive account of relevance feedback techniques. Combinations of clustering and relevance feedback techniques can be used. Most of them are difficult to reconcile with Boolean searching, and some, particularly clustering techniques, can impose very heavy computational loads. One feature which most of them have in common is that the resulting display of records will not be in the order in which they occur in the source file, but in order of decreasing similarity to the query — the meaning of "similarity" being dependent on the method used; it is desirable that this has some correlation with probability of relevance. For subject search queries containing three or more words, Okapi displays records in an order which reflects their degree of similarity to the query (Section 6.6). It does not provide the user with any direct help in modifying the query. CITE (Section 2.4.4) does use relevance feedback, but uses it in a way which requires the user to play a role which is much more active than merely answering "yes" or " n o " to questions about the relevance of displayed records. There does not appear to be any other

9. Conclusions and recommendations

165

OP AC which makes any significant use of unconventional IR techniques.
RELEVANCE FEEDBACK AND CLASSIFICATION

The central problem is that of finding not only those records which are indexed by some or all of the terminology of the query, but also other, related, records. Relevance assessments are easy to obtain — they give the user a sense of useful and positive involvement and need only require a "yes" or " n o " response — and they must often provide useful additional terminology which can be used to revise the search formulation. In many catalogues there are no assigned subject headings, but almost all contain classification numbers, and it is almost self-evident that, among records classified similarly to known relevant records, there will often be some which are also relevant. It is equally self-evident that the use of classification alone will often decrease the precision of a search. This effect may be minimised by using a combination of classification together with free and/or controlled terms from relevant records. It may also be possible to exclude records on the grounds that they contain terminology in common with record(s) judged non-relevant. Some real subject searches from the Okapi logs have been repeated under experimental conditions, and in a considerable proportion of these searches additional relevant records were indeed obtained simply by displaying records classified in the same area as a record which has been judged to be relevant, and this can be done with a minimum of user expertise. It is more or less equivalent to the process of bookshelf browsing, with the advantage that all the books are there, and one can browse many shelves without moving from the terminal. There is little doubt that recall certainly, and possibly precision, could be further increased by adding terms such as title words from relevant records to the query, and perhaps also some account could be taken of the terminology and classification of records judged wow-relevant. A course of research along these lines has been proposed in some detail in [11]. 9.4.2 On data input by the user: spelling and keyboarding

A large proportion of users' time at the OPAC terminal is spent keying in search statements; many of these are amended by the user before the search is started, but many still contain spelling and keying mistakes/])
(1) The Okapi transaction logs suggest a mean effective keying speed of well under one keystroke per second. In a sample of 220 consecutive subject search statements 24 contained one or more spelling mistakes. The mean length of the subject search statements was about 20 characters.

166

9. Conclusions and recommendations

This reduces the cost effectiveness of OPACs, and their attractiveness to users, and is one of the reasons why one probably needs more OPAC terminals than microfiche readers to service a given population. It is ironic that we are entering the age when computers are becoming everyday tools at the same time as the concept of correct spelling is evaporating and we seem to be reverting to the condition which obtained before the eighteenth century. It is very easy for the human eye and brain to cope with orthographical variation, but quite beyond the current state of knowledge to write computer systems which have the same tolerance. For OPAC use, misspelling is of far lower importance with catalogues in which items are found only, or mainly, by browsing (that is, in the older, pre-coordinate, OPACs), than it is with keyword, post-coordinate catalogues. This is certainly a feature in favour of the pre-coordinate OPACs. On the other hand, browsing is time consuming and tiring, so there is no doubt that any OPAC ought to lead the user straight to the relevant item(s) whenever possible. Much research has been done over the last twenty-five years on computer programs for the automatic or semi-automatic correction of misspellings, and, although there are no methods which give results comparable to those obtained by an expert human, there are a number of procedures which are good enough to be worth incorporating in an OPAC. The more "intelligent" methods, such as that of Yannakoudakis and Fawthrop [12,13] need large amounts of memory and processing power, but methods which rely on storing in an index more or less "fuzzy" representations of words and names are quite feasible with the resources which are currently available to libraries. An example is the n-gram similarity technique given by Freund and Willett [14] and used by Porter [15] in a computerised museum catalogue. The user interacts with such procedures by choosing from a list of similar terms. At the most rudimentary level, it should be noted that if a word entered does not appear in a large catalogue, then there is quite a high probability that the word is misspelt, and the system can give a clear but neutral response like: "econmics": nothing found which suggests to the user that s/he may have made a mistake (without making the system look idiotic when the word in question is correctly spelt).

9. Conclusions and recommendations

167

One of the proposals for further development of Okapi suggests the investigation of the effectiveness of a degree of fuzzy matching of personal names and of words in subject search queries [16]. On the use of keyboards, it is probable that not much can be done except by making OPACs which enable and encourage users not to type more than is necessary to retrieve what they are looking for (this applies particularly to specific item searches). It seems fairly clear that no alternative keyboard would give a significant improvement over the standard Sholes (QWERTY) layout [17]. On alternative methods of data input, the drawbacks of touch screens have been mentioned in Section 7.3.2. Input devices like mice (mouses}), lightpens and joysticks cannot do anything, in the catalogue context, which cannot be done with touch screens. In ten or twenty years we may have voice input systems adequate for the enormous vocabulary and range of users which a catalogue must handle. Handwriting recognition is not so remote, but does not avoid spelling problems. 9.4.3 On OPAC output

OPAC searches which involve browsing records or index terms would be both quicker and more pleasant if the clarity of the display were improved and the amount of data on the screen increased. Conventional VDTs have very poor readability compared to printed text: to obtain comparable readability it is necessary to use 60%-85% white space as opposed to 20%-60% for printed material (Section 7.4.3). The poor quality is caused by low character definition, limited character set, little or no choice of font, character size and spacing and the paucity of ways of emphasising and distinguishing between elements of the display (usually limited to two degrees of brightness and sometimes an underlining facility or the use of inverse video). Microfiche material suffers some, but not all, of these drawbacks. Display devices of much higher quality are now obtainable, and are becoming less expensive. The greatest improvement in scanning speed and comfort will probably be obtained by providing much larger, vertically oriented, black-on-white screens. If the definition is high enough, an A4 sized screen can carry about 70 lines of 80 characters, and could be used to display up to about 30 brief bibliographic records or two to four fairly full ones. (Most existing OPACs show six to ten brief records per screen, or at most one full record.)

168

9. Conclusions and recommendations

A typical browsing display (unless it is an index display) needs one disc access for each record, so an OPAC equipped with large, high-definition output devices imposes additional disc access, data transmission and memory loads, and so designers must take account of these heavier storage and transmission demands. OPACs of the near future must also be able to display at least the full MARC character set, as do some cataloguing systems. It may then become inadvisable to use one of the standard character codes (e.g. ASCII or EBCDIC) for the internal representation of characters, but rather to re-code so that, for example, e-acute comes between e and f in the collation sequence. 9.4.4 On user interaction in general

Hildreth has urged that an OPAC should be ''adaptive, collaborative and Socratic" in its interaction with users [18]. It must be readily usable by people with varying degrees of experience, proficiency and aptitude, without only catering for the totally unskilled. It should guide users with any amount of proficiency in obtaining good results and in learning to make better use of the system. No existing OPAC approaches this ideal interaction, partly because we do not know how to implement it, and partly because of the demands on computing resources and human design and programming resources which it would make. Of existing OPACs, almost all are fairly readily usable by fairly unskilled people. Some achieve this by offering two levels of interaction, others have only one, which is aimed at inexperienced users. At the lower (or only) level, there are few OPACs which offer sophisticated IR features. (As usual, CITE (Section 2.4.4) is an exception, but is not suitable for very unproficient users.) There has been no serious attempt to make an OPAC which adapts, actively or passively, to different types of user or to the course of an indivdual search. Such an OPAC would be an example of the type of "highly interactive" or "adaptive" system discussed by, among others, Hayes in [19] and Innocent in [20].
HIGHLY INTERACTIVE OPACS

Marcus and Reintjes [21], cited by Hildreth in [22], list the following dimensions along which interaction can vary:

9. Conclusions and recommendations 1.

169

Verbosity I terseness in the number and comprehensiveness of messages.

2. Instructional I service: system messages and interaction in general can be set so as to be more or less didactic. 3. Interpreted/strict: in the OPAC context, interpreted mode would extend a search by using semantically or morphologically related terms (Section 9.4.1), whereas strict would keep precisely to the format and terminology of a search statement. 4. Automatic Iassisted: the degree of user involvement, the extent to which the system requests user guidance. 5. Hidden/expository: the extent to which the system reveals its workings. There could be continuous variation along each of these dimensions, and it need not be constant throughout a session or even a search. The verbosity I terseness scale is probably the one which needs the most urgent attention. One of the major problems with interactive systems is that of drawing the user's attention to vital information at the time it is needed. Experience has shown that casual OPAC users (and many catalogue uses are and should be casual) take a minimum of notice of messages on the screen, often reading a message only after several failed searches. Further, after gaining a little experience with a system, users commonly assume that they know all they need to know, and continue to make searches which are less efficient than they should be. Hence it is important that messages and prompts should be limited to what is necessary for a particular user at a particular point in a particular search/ l) It is therefore necessary to reduce messages to a minimum (one message, two to four alternative actions depending on how concisely they can be
(1) There is a connection here with the poor readability of conventional VDT displays, mentioned in Section 9.4.3. Without blinking or flashing, which is rather drastic, or making a noise, which is unpleasant, there is little which can be done to attract the attention of a user. However, improving the readability and presentation of displays would undoubtedly alleviate the problem somewhat.

170

9. Conclusions and recommendations

explained) — this should be obvious from analogy with (spoken) dialogues between humans in similar types of situation (helping, explaining, advising, searching). The whole process must be highly interactive: a brisk exchange of brief messages. The system must know what to do, and when it does not know it must seek elucidation from the user.
EXAMPLES

"There are a lot of Smiths — can you give an initial?" "6 under realism, 15 under naturalism, nothing under both: please choose the one you want" "There's nothing very similar. Can you think of something less specific?" (After a longish pause) "Press the HELP key for advice on how to enter your search" (User slowly typing long title) "There's usually no need to enter the whole title" To achieve this, the system must be able to infer with high reliability what the user currently needs to know, what choices of action are sensible at this point. This would seem to suggest some sort of artificial intelligence application, but the term seems to have become both too general and too specific. All it really needs is the application of artificial common sense. Knowledge of how to make such highly interactive systems is derived primarily from a study of what people do when using OPACs (Section 9.4.6), with particular attention being paid to searches which failed or only partially succeeded. Some preliminary work towards the evolution of a highly interactive OP AC has been proposed in [23]. 9.4.5 On bibliographicfilesfor OPACs

OPACs vary widely in the structure and complexity of their bibliographic files. Some use a full MARC record (doubtless with the tags embedded and the directories removed), some, like many of the OPACs derived from circulation systems, use an overly abbreviated record; others, like Okapi, retain most of the information from the MARC record but discard a good deal of its structure.

9. Conclusions and recommendations
IMPLICATIONS FOR REMOTE ACCESS

171

There are two ways in which users of a library could obtain remote access to the collections of other libraries: one method is to access the remote OP AC, using local terminals; the other is to use the local OPAC to access the remote files. The first method can be used now — there are a number of OPACs both in the US and the UK which can be accessed through public networks. Access to a remote OPAC entails either that the OPAC be instantly usable by those unfamiliar with it, or that there should be an interaction mode which is standardised. There have been suggestions in the US that there should be some attempt at drawing up a standard command language for OPACs. However, one of the main burdens of this report is that the very concept of a command language is not appropriate for public access (Section 7.4.1), and the approach should rather be to make the OPAC transparent (in the sense of making the interaction self-evident). The alternative is to standardise OPAC files and their indexes within the participating libraries. This condition already holds for the files of libraries whose OPACs use a full MARC structure, but it seems too much to hope that a satisfactory standard for indexing would be achieved. A compromise which seems just conceivable is that groups of libraries might be able to agree on a simpler standard, derivable from MARC, for OPAC files and indexes. The Centre for Catalogue Research report [24] showed convincingly that almost all catalogue uses would be satisfied by a record very much simpler than the full MARC record (indeed, by a record which is even simpler than the Okapi record). A further point which has to be considered in this connection is that many libraries would wish to be able to output records in exchange format for use by other libraries. The Okapi source file is an example: although it can be used to generate catalogue records adequate for the requirements of most library users, it would be impossible to produce from it anything resembling a subset of a MARC record.
ENHANCING THE CONTENT OF BIBLIOGRAPHIC RECORDS

Certainly, few libraries need full MARC records for any purpose except that of exchange, but it may be that there are areas in which the structure of standard bibliographic records could even be elaborated.

172

9. Conclusions and recommendations

The CLR survey [25] found that many OP AC users would like periodical articles to be accessible. MARC does not specify a method for cataloguing these. It is possible to catalogue journal issues as records with analytical entries, within the MARC structure, and this has been done by some libraries in the UK. Although the UK MARC Manual says that "analytical entries are made for up to three works contained within an item" [26, page 5/50], there is no reason for abiding by this restriction. Rule 1.7B18 and Chapter 13 of AACR2 apply [27,28]. Apart from the matter of periodical articles, there are several ways in which additional subject-rich entries could be provided, some for display, others to provide index terms for OPACs. Some of the MARC notes fields, particularly 505 {contents) and 513 {summary), could be used to improve both access and the informativeness of displayed records. Apart from its use in analytical cataloguing, 505 can be used to contain information from contents pages, thus enhancing both access and informativeness as in the "BOOKS" project [7]. The MARC 513 field could contain abstracts, which would be particularly useful for periodical articles (since abstracts are often available). There is also an index terms field (695), reserved for future use.
SIMPLIFYING BIBLIOGRAPHIC RECORDS

One recommendation for the simplification of MARC is for the elimination of a feature which must have irritated many users — statements of responsibility. It is possible to repeat the names in these in 700, 710, 711 (name added entry) fields, but these fields do not allow at present for recording the role of the person or body named except in the case of collaborating author; AACR2 [27, Rule 21.0D] allows the optional addition of designations of function, but this has not been implemented in UK MARC. There are seven spare values (3-9) of the second indicator which could be used as function designators (translator, illustrator, etc.). If this were done, those libraries which wished to display something in the nature of a statement of responsibility could still do so, while avoiding the present duplication of data.
T H E "MAIN ENTRY" CONCEPT AND THE ORDER OF RECORDS IN OPAC DISPLAYS

Going a little further, what would be the implications of eliminating the concept of main entry} Except for some of the more primitive ones, OPACs do not, and arguably should not, suggest to the user that there is

9. Conclusions and recommendations

173

one physical sequence of records. There is evidence that users who have slight acquaintance with OPACs feel that there is a sequence, but it seems likely that OPACs will come to be regarded as entities which behave more like reference librarians than catalogues. In any case there should be no single fixed sequence, but an indefinitely large number of apparent sequences which depend on the search being made. (How many users of online reference retrieval systems regard them as providing access to ordered files?) Apart from the attitude of users, the elimination of the main entry concept together with the statement of responsibility would remove a burden from cataloguers by enabling them to enter all names only in 700 and 710/711 (or 100 and 110/111) fields, instead of some in 7nn, some in Inn or 245 and some in both. Nevertheless, the question of order in OPAC displays does need attention so long as considerable numbers of records may result from a search. There is no space here for a full discussion of this contentious topic, but one or two of the issues are touched upon. Clearly, the needs are very different in subject and in specific item searching. The main function of the apparent ordering of a browsing display is so that the user knows whether to go forwards or backwards, and, most importantly, when to give up on the grounds that the sought item is not there (think of searching a random telephone directory). Traditionally, there were elaborate filing rules for catalogue cards, and for dictionaries and street indexes and the like. Some of these rules depended on knowledge which it is difficult or impossible to incorporate in a computer program (even to make " 3 0 " file with "thirty" is a major piece of coding). These rules are not usually known precisely by users (does D e a t h come before or after D e - a t h ? ) , but there is enough consensus that it is usually possible to reach approximately the right area in whatever is being searched, and then to look at every entry. In OPACs and other IR systems display order is sometimes imposed by the fact that the easiest way to add new records to a file is to put them on to the physical end of the existing file, which leads to records with identical keys being output in order of date of addition to the file (preferably reverse order, since new material is usually in heavier demand than old). Those who are worried by an apparent lack of order in, say, a display of records of books by a prolific author, will doubtless be satisfied by a form

174

9. Conclusions and recommendations

of indexing which concatenates a portion of the title onto the author key (as in the first example in Section 5.3.1); but the systems which use this type of indexing are primitive ones where there is no concept of "exact match": items can only be found, as in a conventional catalogue, by scanning. In any case, searches for works by an author form only a small proportion of catalogue uses. In specific item searching browsing is unnecessary when there is an exact match: being forced to browse is a waste of the user's and the system's time (Section 7.3.1). In subject searching, on the other hand, browsing is essential, but there is only one satisfactory way of ordering the displayed records: by probability of relevance [29]. 9.4.6 On the evaluation of OPACs and the study of their use

There is far higher variability among OPACs than there is among conventional catalogues. There is no received opinion about what constitutes a good OPAC, although the principles set out by Hildreth [22], and many of the recommendations given above, can hardly be called contentious. There is no serious theoretical difficulty in making a comparison between two catalogue systems, provided that they serve the same user population and access the same collections): the comparison by Siegel and others of CITE with ILS in the National Library of Medicine demonstrated this [30]. As a step towards the comparison of a wide range of OPACs, there is something to be said for the use of a set of standard searches, carried out by an experienced researcher. However, the purpose of evaluation must be to help us to design better catalogues, and there are far too many variables, and inter-dependences, for any highly objective and analytic evaluation to be helpful or even possible. It follows that the process of producing improved OPACs necessitates studying what features of existing OPACs appear to be particularly good, or bad, and why they are so. Some OPAC properties can be evaluated without regard to the user interface — these are what we have referred to as OPAC search functions (Sections 2.7.1 and 8.1.2). Available features can be tested without regard to ease of use or whether they would be used by real searchers. Such evaluation forms an essential part of the development process for any OPAC, but since the user is an inseparable part of an interactive system, the most important evaluation must take account of users' needs and behaviour. This is not to say that every evaluation experiment has to have real users: an experienced experimenter will have a very good idea of how

9. Conclusions and recommendations

175

users would react at various points during their interaction with the OPAC. So far as possible an OPAC must function efficiently given the often conflicting requirements of different types of users and needs. Hence some sort of classification of user types is important: apart from demographic data (Section 8.1.2), users can be categorised by their degree of familiarity with the OPAC under consideration, by their exposure to interactive computer systems in general and by their academic level (Section 7.2). A classification of search types (user needs) is also necessary.
WHAT SHOULD BE EVALUATED?

The following can be evaluated with respect to different types of users and searches: (1) Speed and effectiveness — in other words efficiency. It is not even known how OP ACs compare with book, card or microform catalogues (except that OPACs are generally more popular). In particular, there is an urgent need for a comparative study of catalogue use in a library which is in the process of changing from a conventional catalogue to one of the more advanced OPACs. Ease of use and of learning Irelearning, and how it relates to the features available and the ways in which they are offered, and methods of providing help and advice, of guiding the user through the search process, both within the OPAC and externally. Are there features which are not used? If so, is this because they are not useful, or because they are too difficult to use? What effect does experience of one OPAC have on users' behaviour with another? User input. To what extent, if any, are devices other than keyboards useful? What input formats are most efficient and acceptable? Should subject searches be entered in ''natural language", or is it feasible to attempt to obtain search statements in a form reducible to some kind of Boolean expression? Display content and presentation, and questions of order, and of the provision and use of browsing facilities.

(2)

(3)

(4)

176

9. Conclusions and recommendations

EVALUATION METHODS AND SOURCES OF DATA

The most easily obtainable data are from transaction logs, produced automatically by the search system. Most OPACs incorporate some degree of logging, but its nature and comprehensiveness vary widely. A minimum requirement is that the logs should contain sufficient information for searches to be repeated by an experimenter. It is highly desirable also that they lend themselves readily to automatic analysis — that their format is such that it is easy to write computer programs to analyse the logs in ways which were not necessarily foreseen. They should also contain timings of every phase in the interaction, so that, for example, the degree of familiarity of a user can be estimated from the amount of time spent reading instructions. One of the shortcomings of automatic logging is that, with an OPAC, it is often not possible to determine session boundaries (that is, the point at which a specific user starts and finishes a session — see Section 8.2) because it is not reasonable to expect users to sign on and off as they do in some other types of interactive system. There is much that cannot be deduced from even the most detailed of transaction logs. This includes data about users' attitudes, needs, degree of satisfaction and understanding of the system, as well as demographic data (although some systems — Paperchase is an example — try to obtain some of this information by asking the user to complete an online questionnaire at the end of a session). Hence automatic monitoring has to be supplemented by observation and by structured or unstructured interviewing. There is also a place for the use of questionnaires — the one used in the CLR study [25] was fairly comprehensive, although rather "closed" (Section 8.1). 9.5 Concluding remarks

The key problem in OPAC design is that of making sophisticated retrieval facilities available, while retaining ease of use and reducing learning effort to a minimum. It is quite easy to achieve ease of use — we are certainly not under the illusion that Okapi is the only OPAC which is usable at sight. It is less easy to make a retrieval system which compensates for searchers' lack of knowledge of the ways in which items in its databases are described, and outputs results in order of probable relevance to the request. It will be extremely difficult to produce a system which achieves both goals, although we do not believe that this is unattainable.

9. Conclusions and recommendations

177

Such an ''ideal" retrieval system would require very large investments in file design and access methods, in functional design and in the investigation of modes and structure of user interaction procedures. Some of the methods for approaching this goal have been outlined in Section 9.4. They require the use of advanced hardware (large amounts of core memory, high data transmission speeds, high quality display devices), and the application of software techniques which have evolved since the advent of personal microcomputers and in some of the applications of artificial intelligence. Above all, they call for the study of users' needs, and for a continuous process of evaluation and feedback. To date, no OP AC or other reference retrieval system even approaches this ideal, and it is clear that few, if any, organisations in the library world have the resources to make the required major investment. New OPACs are appearing almost weekly, as part of integrated library systems. Since the market is highly competitive, and the OPAC has only recently become one of system purchasers' requirements, many of the suppliers add "public access" to their systems hastily and almost as an afterthought. If library management, from national libraries downwards, would come to realise the increase in the use of library materials which would result from the introduction of catalogues which actually help most users to find what they are seeking, and that the systems they are buying now may well be totally obsolete in five years' time, it might be that automation suppliers and national or international bodies would cooperate in helping to fund research into the development of the catalogues of the future. References 1 2 3 4 Persky G and others. A Geac local area network for the Bobst Library. Library HI TECH (6), 1984, p37-45. Markey K. Subject searching in library catalogs: before and after the introduction of online catalogs. OCLC, 1984. Kaske N K and Sanders NP.^4 comprehensive study of online public access catalogs: an overview and application of findings. Final report to the Council on Library Resources. Vol III. OCLC, 1983. Walker S. The free language approach to online catalogues. In: Keyword catalogues and the free language approach. Edited by Philip Bryant. Centre for Catalogue Research. Bath University Library. 1985.

178 5

9. Conclusions and recommendations Bates M J. Factors affecting subject catalog search success. Journal of the American Society for Information Science 28 (3), 1977, pi 61169. Hildreth C R. Pursuing the ideal: generations of online catalogs. In: Online catalogs, online reference — converging trends. Proceedings of a LITA Preconference, June 23-24 1983, Los Angeles. American Library Association, 1984. Atherton P. Books are for use: final report of the Subject Access Project to the Council on Library Resources. Syracuse University, School of Information Studies, 1978. Markey K. DDC as a library user's tool. MARC Users' Group Newsletter. Issue 85 No 1, March 1985, pl9-26. Markey K. Subject-searching experiences and needs of online catalog users: implications for library classification. Library Resources and Technical Services, January-March 1985, p34-51. Harper D J. Relevance feedback in document retrieval. PhD Thesis, University of Cambridge, 1980. [Walker S ]. On the use of relevance feedback and classification to improve subject access fin an online catalogue]. Draft proposal submitted to the British Library Research and Development Department. 1984. Yannakoudakis E J and Fawthrop D. The rules of spelling errors. Information Processing and Management 19 (2), 1983, p87-99. Yannakoudakis E J and Fawthrop D. An intelligent spelling error corrector. Information Processing and Management 19 (2), 1983, plOl-108. Freund G E and Willett P. Online identification of word variants and arbitrary truncation searching using a string similarity measure. Information technology: research and development 1 (3), 1982, pi 77187. Porter M F. Information retrieval at the Sedgwick Museum. Information technology: research and development 2 (4), 1983, pi 69186. [Walker S]. Improving access in an online public access catalogue by automatic word stemming, spelling correction, and limited synonym generation. Research proposal submitted to the British Library Research and Development Department. April 1985. Norman D A and Fisher D. Why alphabetic keyboards are not easy to use: keyboard layout doesn't much matter. Human Factors 24 (5), 1982, p509-519.

6

7

8 9

10 11

12 13

14

15

16

17

9. Conclusions and 18

recommendations

179

19

20 21

22 23

24 25

26 27 28 29 30

H i l d r e t h C R. [Users and online systems: the user interface]. Paper given at a conference "Online public access to library files", Centre for Catalogue Research, Bath University Library, 3-5 September 1984. H a y e s P J and R e d d y D R. Steps toward graceful interaction in spoken and written m a n - m a c h i n e communication. International Journal of Man-Machine Studies 19, 1983, p231-284. I n n o c e n t P R. T o w a r d s self-adaptive interface systems. International Journal of Man-Machine Studies 16, 1982, p287-299. M a r c u s R S and R e i n t j e s J F. The networking of interactive bibliographic retrieval systems. M I T Report E S L - R - 6 5 6 . N T I S PB252407. M a r c h 1976. H i l d r e t h C R. Online public access catalogs: the user interface. O C L C , 1982. [Walker S]. The design and development of a self-adaptive user interface for an online public access catalogue. Draft proposal submitted to the British Library Research and Development Department. 1985. S e a l A , B r y a n t P and H a l l C. Full and short entry catalogues. Centre for Catalogue Research. Bath University Library, 1982. M a r k e y K. Online catalog use: results of surveys and focus group interviews in several libraries. Final report to the Council on Library Resources. Vol II. O C L C , 1983. UK MARC Manual. British Library. Bibliographic Services Division. 2nd ed., 1980. Anglo-American Cataloguing Rules. Library Association, 2nd ed., 1978. G r e d l e y E (Polytechnic of N o r t h L o n d o n ) . Verbal communication, M a r c h 1985. R o b e r t s o n S E . T h e probability ranking principle in IR. Journal of Documentation 33 ( 4 ) , 1977, p294-304. S i e g e l E R et al. A comparative evaluation of the technical performance and user acceptance of two prototype online catalog systems. Information Technology and Libraries 5 ( 1 ) , 1984, p35-46.