143

8
8.1

Okapi in use
Introduction

'' If information systems are to be truly responsive to users' needs, change itself must be considered in the process of system design. Systems design can no longer be viewed as a one-time effort resulting in a static design that is unchanging for the operational life of the system. ... Evaluation is defined as the process of analyzing the functioning and/or usage of a system so that decisions can be made concerning the effectiveness of the system in satisfying its design objectives." [1]. 8.1.1 Brief literature review

The most important OPAC use study is the CLR-funded evaluation project in the U S ; for a short description of the scope and objectives of the study see [2]; for an overview of the findings see [3]. The CLR evaluation gathered three types of data: transaction log analysis, "focus group interviews" and questionnaires. This study was carried out on a very large scale (public, university, government and national libraries, with questionnaires administered to thousands of users). The results of the log analysis are largely quantitative and statistical and are presented in the form of many diagrams, tables and cross-tabulations. The interviews focused on specific groups of people, such as cataloguing staff or young library users, and the interviewer made them talk about the problems they had with OPACs. This resulted in a more qualitative type of data. There were two questionnaires, one for users and one for non-users; these are extremely detailed and they consist mainly of "closed" multiple-choice questions. Apart from this very large-scale study, there are also a number of smallscale usage studies, such as the following: a comparison between two OPACs [4], or more [5], the study of transaction counts of one OPAC over a long period of time [6], [7], comparisons of card and online catalogues in the same library [8]. For a good review of OPAC use studies see Chapter 2 of [9]. A recent paper of Cochrane and Markey [10] recommends which methods to employ when studying the use of OPACs.

144

8. Okapi in use

The next section analyses what is generally meant by the study of the use of OPACs and attempts to classify different approaches. 8.1.2 What can be evaluated and why?

The performance of OPACs needs to be evaluated. It can be seen from a library use point of view: for example by comparing the use of an OP AC and a COM catalogue in the same library, to see which one satisfies users' needs best. A designer and a library manager might be interested in the comparison of the use of several OPACs for different reasons: the former in order to improve OP AC design, the latter in order to buy the best system available. Both might want to know which ones are preferred by the users, which features are essential, which systems provide more sophisticated features, etc. OPACs can also be seen from an information science point of view: their major characteristic is that they are IR systems used directly by end users; evaluating them is the best way at present of gathering information on (a) the information searching behaviour of a large user population and (b) the usefulness and effectiveness of traditional IR techniques such as Boolean searching, with a non-trained public and a wide range of information needs. Studying these last two aspects helps to elucidate the issues raised by evaluation from the other viewpoints: if we know more about users' searching behaviour and how current IR systems respond, then better systems can be designed for more people. There are three main aspects of OPACs which can be evaluated. (1) The technical performance, which has to be taken into account when comparing two OPACs. One can examine, for example, the response time as a function of the number of users and stations, or the way response time varies with the type of search done. The IR performance, for example: the frequency of matches/no matches by type of search, the number of hits by type of search, phrase-matching success and recall and precision. User behaviour, because system performance is linked to system usage. This includes studies of users and of use: (i) Identification of user profiles which should include: information on users' background (demographic, academic), on their

(2)

(3)

8. Okapi in use

145

attitudes to and experience of computers and libraries, on their information needs. User search patterns and interaction behaviour should be analysed and correlated with these profiles. (ii) User search patterns are, to a certain extent, defined by the system users are interacting with. But there have been some attempts at identifying some general/common user search patterns with OPACs [1], [9] or [11]. These search patterns include, for example, the frequency of single versus multiple searches per user session, the time spent doing various types of search, the rate and locus of error, the frequency of unsuccesful searches in which users re-enter or revise their query, etc. User interaction success. This can be collected from the users themselves (their subjective opinions about ease of use, learning, re-learning, etc.) and by using an experimenter who knows the system and its limitations (for example, to repeat logged searches). The study of user interaction includes the following two aspects: (a) dialogue success, e.g. which system capabilities are too difficult to use, the ease of input of authors, the clarity and understanding of instructions, messages and help, which display formats are un/acceptable (content and readability), keyboard problems, etc. IR success, which is obviously linked with (2) above, but is seen here more from the user's point of view. It covers, for example, how well users' queries are matched by the system, whether users' searching behaviour is supported or hindered by the system, etc. Users can also be asked their opinion about the relevance of the records retrieved.

(iii)

(b)

8.2

Methodology

It was impossible to carry out a comprehensive and detailed evaluation study of Okapi given the staff and time constraints. Nevertheless, it was important to gather some information on the live use of Okapi, especially since ease of use was the primary aim. Also, implementing a "naive" interface (Section 7.4.1) leaves the possibility of introducing more

146

8. Okapi in use

sophisticated features at a later stage, and seeing how users react to them. 8.2.1 Automatic logging

First of all, it seemed relatively simple and worthwhile to have some automatic monitoring of usage. Since Okapi is implemented locally, it is easy to run a logging program as and when wished, without any extra cost. The only effort was in designing and writing logging procedures; logging is then completely automatic. As can be seen in the sample printout in Appendix 9, Okapi logging procedure makes quite a detailed record of the searches done (see [11] for examples of other logs). Searches can be repeated easily by an experimenter, so that, for example, the records seen by the user can be checked for false drops. All keys pressed are recorded together with the time taken by the user and the system for various tasks (Section 9.4.6).
SESSION BOUNDARIES

The user session is the natural evaluation unit. As was mentioned in Section 7.5.1, it was hoped that the use of the BLACK key (labelled 'Tress when finished") would delimit user sessions automatically in the log file. Unfortunately, this was not successful: the students used the BLACK Key for other unintended purposes, such as getting out of an option (in particular to make their previous input disappear from the screen) and starting again from the beginning, rather than using the RED key, which takes the user back one step only.
LOG ANALYSIS

Nevertheless, it seemed worth analysing the logs, even without session boundaries. Some analysis was done on 3912 searches logged between 28 November 1984 and 22 January 1985. The first obvious use of the automatic logging method is to give indications of the user search patterns mentioned above in Section 8.1.2 (3) (ii): the logs were analysed by computer to extract some statistical information, such as the proportion of subject searches, the average number of words in subject input, etc. The logs can also be used for judging the IR performance (Section 8.1.2 (2)) and some programs were written to count, for example, the number of successful and unsuccessful searches and number of matches, according to the number of words in subject search statements. The main results are given in 8.5.

8. Okapi in use

147

Information on the user interaction success (Section 8.1.2 (3) (iii)) can also be found from the logs. But no program can be written to do this, partly because of the impossibility of recognising user session boundaries automatically. It has to be done by hand: a human reader can more or less guess the session limits, and reading through the logs is extremely rich and valuable; but the information gained is very difficult to formalise. 8.2.2 Observation and structured interviews

Automatic monitoring provides only a limited amount of information. One cannot rely solely on a logging procedure to study the use of an interactive system. The identification of user profiles and the interaction success mentioned in Section 8.1.2 (3) (i) and (iii) respectively, cannot be recorded by a logging procedure. Online interviewing has been used in OP AC evaluation (for example with Paperchase [12]). This technique was considered (making the system ask questions such as "have you used Okapi before?", "are you a Social Sciences student?") for collecting information on the users themselves (i.e. (3)(i) of Section 8.1.2); but this is both unreliable and tiresome for the users. A human intermediary seemed unavoidable. Human intervention is essential, for example in analysing users' conceptual processes if they discuss their search strategies. Several methods could have been used to gather different kinds of information: interviewing (the talk-aloud technique has been used while students search, but this may interfere with the activity being carried out), observation (without any interference from the observer), survey (of users and non-users), questionnaires, group interviews (with students and staff), controlled experiments.
OBSERVATION

The observation method was chosen for three reasons: it was useful to get some feeling of students' behaviour in a natural environment; it involved comparatively little work; some human marking of user sessions was necessary anyway, because it was not clear if the logging procedure would be reliable enough to delimit user session boundaries. The simplest procedure was to have the experimenter sitting at a desk not far from the Okapi terminal, recording the beginning and end of user

148

8. Okapi in use

sessions and recording what was going on, onto an observation sheet, which would later be compared with the printed log (and the answers to the structured interview, see below). The experimenter was extremely familiar with Okapi, so was able to follow what the user was doing without being too near nor seen (in most cases) and without interfering. The purpose of the observation sheet was to keep track of activities extra to those recorded by the logging; i.e. problems with coloured keys and keyboard, hesitations, looking at papers, writing, waiting, external signs of the interaction (sighing, making loud comments, touching the screen, shouting...), talking to other students, asking for help from other students or from the staff and sometimes from the experimenter. This corresponds to the evaluation of the dialogue success mentioned in Section 8.1.2 (3) (iii) (a). No results of the observation as such will be presented in this report.
USER SESSIONS

The observation allowed 96 user sessions to be clearly isolated, between 22 November 1984 and 9 January 1985. A manual analysis was performed on the logs of these user sessions; it gave more detailed results than the automatic analysis of the 3912 searches, for example: the spelling mistakes in different types of input (which a program could not extract). The user session results are given in Section 8.4.
STRUCTURED INTERVIEW

As one member of the team would have to spend some time on site observing users, a short questionnaire was designed to get some additional information (on students' background and opinions, see Section 8.1.2 (3) (i)). The procedure chosen to administer the questionnaire was similar to a structured interview: the students were not asked to fill it in themselves but the observer asked the questions and took down the answers. The general approach was that students should not have to choose among predefined categories, so that they would give more natural and spontaneous answers and ideas. Section 8.3 contains the main results of 70 structured interviews.
CONTROLLED EXPERIMENTS

It was also intended to set up some controlled experiments. It was planned, for example, to present five or six pre-selected searches of

8. Okapi in use

149

different levels of difficulty in randomised order to students (with enough students to do all permutations) and see how different types of users cope. Several aspects of search difficulties could have been isolated and tested: problems with the input of authors (personal, corporate), problems of no match/lots of matches in subject searching (how to narrow/broaden) and known item searching, problems of phrase/word matching. Other areas of interest were considered, such as the readability and clarity of different types of screens and record displays (including the matter of record ordering). But these experiments would have involved too much work in design, organisation and programming to be carried out within the allocated time. To summarise, the data collected during the evaluation phase of this project consist of: 70 structured interviews, 96 user sessions and 3912 logged searches. It would have been very interesting to try to correlate the interviews (and perhaps also the experimenter's observations) with the corresponding user sessions, especially since these two types of data are rarely collected together. But this is not a simple task and would have been very lengthy. 8.3 Structured interview results

The evaluation took place in the reference area of one site library of the Polytechnic: the Social Sciences, Business Studies and Communication Library, in Riding House Street. This area of the library is usually quite crowded and there is not much space. An effort was made not to interfere too much with the students and with staff activities. One microfiche reader was left in its normal place, so that students would still be able to find and use their usual catalogue. The COM catalogue used at PCL consists of two microfiche sequences, author-title and classified (Dewey numbers). There is a printed subject index available for consultation. The second microfiche reader situated next to the first one was replaced by an Okapi terminal; a poster above the terminal explained that Okapi was an experimental computer catalogue, that anybody could use it, and that all comments were welcome. There was also a suggestion book. The data gathered are of three types:

150 (1)

8. Okapi in use students' academic background and experience in the use of computers, keyboard, catalogues and computer catalogues — see Appendix 6 (Part 1). users' reactions, opinions and comments on Okapi — see Appendix 6 (Part 2) for quantitative data and Appendix 5 for qualitative data (list of actual comments). questions about their search: what they were looking for and if they found it — see Appendix 6 (Part 3).

(2)

(3)

Some cross-tabulation of the answers was also done and this is in Appendix 6 (Part 4). 8.3.1 Users' background and experience

The full results are in Appendix 6 (Part 1). Among the thousand users of the site library, 70 students were interviewed. More male (41) than female (29) students were observed using Okapi and interviewed. On the whole, it is a young population: 50 students were between 19 and 25 years old; there were more third year students than first or second years. A large majority of the students were on social sciences courses. Most students interviewed were first-time users of Okapi, in fact of any computer catalogue. Their previous experience of catalogues was chiefly of the COM catalogue. Although 59 students had been in contact with a computer before, 39 of them did it through a "quantitative methods" course where they learn how to run statistical packages on a mainframe. Most of them felt this had not been useful. A few of them had a suspicious attitude to computers in general ("I ignore them and don't like them", "daunting technology"). 8.3.2 Users' opinions of Okapi

QUANTITATIVE RESULTS

A large majority (90%) of the students said they liked Okapi and that they would use it again; see Appendix 6 (Part 2). Four users said they did not like it, but that they would use it again. But a much better indication of ease of use is the following: nearly two-thirds of the students (62%)

8. Okapi in use

151

thought that they would need five minutes or less to feel confident about using Okapi; see Appendix 6 (Part 2).(1) Correlating students' background and previous experience of computers, keyboard and COM catalogue with the fact that they liked Okapi or not is not terribly conclusive; see cross-tabulation in Appendix 6 (Part 4). The main reason is that our sample does not include enough people who did not like Okapi. An interesting result is that whereas 6 of the 41 men (15% ) said they did not like Okapi, only 1 of the 29 women ( 3 % ) said so.
QUALITATIVE RESULTS

The most interesting contusions from users' comments are classified and presented below. Throughout, the numbers between brackets refer to a specific user's comment (see full comments in Appendix 5). Attitude to computers A few students were obviously suspicious about computers when they first started, "it won't bite?" (11), or rather scared, "I didn't dare press the keys, I didn't know what might happen" (3). But some found it "easier than they had expected" (12), thought that it could "be used by people who don't know about computers" (59) or even became quite excited about it: "when are you going to put the whole world in i t ? " (11). One student did the same search three times "to make sure the computer gave the same result" (13). Use of keyboard, typing problems, VDU Typing problems were often mentioned and seem a serious handicap. For example: "If I was good at typing I would use it all the time" (21,43, 57). This led some people to say that because they are so bad and slow at typing, they would rather use the microfiche, e.g. (28, 83). The keyboard gave the impression of being small (1) (although the key spacing of the Okapi stations is standard). One student pointed out that it was "too much trouble to look at the screen and keyboard at the same time" (14). Comparison between microfiche and computer On the whole, students liked Okapi better than the microfiche. There were quite a number of enthusiastic comments in favour of the computer
(1) The question asked ("when will you feel confident about using Okapi?") contains the implicit assumption that users do not already feel confident. In spite of this several people replied they did feel confident now.

152

8. Okapi in use

catalogue: "Brilliant, magic. I'll use it rather than the microfiche" (87), "the microfiche seems dead" (75). Nevertheless, many comments cannot be clearly labelled "for" or "against": students were able to see advantages and disadvantages of each. Physical aspects: first of all, several students said that the microfiche made them "feel dizzy" (24) or "was painful for the eye" (11) and that the computer was "easier to read" (65); this also might be because the microfiche reader was not especially clean; in some cases, this made them prefer the computer (24, 88). Another reason for preferring the computer is that it is "physically less tiring" (1), "less fiddly" (33), "all the books are together" (59) and that the "fiche is always in the wrong place" (57). It seems to be a question of choosing between the lesser of two evils: typing or "fiddling around" with the fiche (7). One student preferred the microfiche, because s/he felt it is "ordered" (6) and another one said that the microfiche "allows you to scan, unlike the computer" (5). Speed, ease of use: many of them thought the computer was faster: "saves time and less laborious. What would take 3 hours with the microfiche takes 20 minutes with the computer" (35). The students who had typing problems obviously felt the microfiche was quicker (4, 28). Many users preferred the computer because it was easier: "just pressing a button, thing comes u p " (84), "more flexible, more pleasant" (81), "more fun" (8), "neat" (19), "more efficient" (30), "more interesting" and "more thorough" (72). Subject versus specific item: quite a number of users felt that the computer was "more useful for subjects than title/authors" (7), that "subject searching is better on the computer, it is like scanning bookshelves" (58), and that if one cannot "remember the title or author of a book" (44) or if the book is not on the shelf or out (36), it is possible to do a subject search instead. They liked being able to retrieve titles and subjects (36), and thought that "the computer gives more useful related information, unlike the microfiche" (48) and (46); one "gets an answer even if rough area" (64, 69), "less chance of missing something" (72). Learning Okapi and ease of use A few students expressed some caution by saying they needed "time to learn it" (3) and that it took them time to get used to the instructions (54). But most users said that the instructions were "clear", "straightforward" (77), "very simple" (55) and that "it is NOT a computer, you

8. Okapi in use

153

don't have to learn it!" (84). It may also be because of the coloured keys, which were very much appreciated (8, 30, 32). Several students suggested they should be labelled as well (14). The only noticeable problem was the Black key; a number of students thought it was not obvious (64, 72). Comments specific to Okapi These are, on the whole, about retrieval problems: title/author false drops (48) which the users generally are not aware of (only one student (25) was); subject search mishaps, in particular when the "hyper-OR" (Section 6.5) does not give anything relevant (1, 11) — (75) also complained about the hyper-OR not "telling when the search is over" — and when searching for names as subjects (88). When these subject search problems arose, students complained about them and said that Okapi "ought to cope with that" (88). Only one student (5) complained about the "too many matches" and "no match" situations and asked to be "oriented towards the library shelves". There was only one rather strange comment about the input of author (14), and somebody asked how to input two authors at once (5). Record displays were mentioned only once and then only as compared with COM records which contain "too many book details" especially "numbers" (5). No student made any comment about having to wait for the hyper-OR to finish; indeed there was no complaint about the response time at all. Suggestions for improvements Apart from the criticisms linked with the IR problems mentioned above, many suggestions are to do with catalogues in general, particularly with the enhancement of subject access, both by providing more access points and by including uncatalogued materials. The following suggestions were made: including the indexes at the back of the books (5), abstracts "to say under which angle the topic is treated" (11, 24), journal articles (29), essays and magazines (41), more keywords per book (81). A few students had a more ambitious approach and asked for access to books from other libraries (33, 74), and to external databases (11). Several people asked for more machines (53), on each floor of the library and everywhere in the Polytechnic (72), and one asked if s/he could "take it home" (74).

154 Users' understanding of and attitudes to the system

8. Okapi in use

Most users did not really understand what was going on. The only exceptions were (46) who understood the number of postings for each word, if not the principle of ANDing two words, and (5) who did comprehend Okapi very quickly and was able to make constructive suggestions ("If there is only one author [with a given surname], there is no point going through author index"). Nevertheless, many students did not seem to be passive: they tried to interpret what happened, for example the author input mentioned earlier by (14), or to explain why the computer did certain things: "I didn't give it enough parameters so the computer couldn't find anything" (23) or "the more specific you are the better" (1). Some of them noticed fundamental IR features: "why does it do words one by one?" (10) or the title/author key principle (25). Although they seemed to trust what the machine told them (75) — or assumed that if they did not find something it was because it was not in the library (14) and put the blame on the library coverage rather than on the computer (83) — they were rather wise: "I might find out the limitations of the computer later" (46); when their search failed they did not blame one element only: "I realised I made a mistake" (12), "I assumed it was because my reading list is very bad". One student (58) gave a fair description of the situation from several angles at once: "difficult to know if I made a mistake or if it is because there are no books in the library ... It is a bit like fishing in the dark ... Opinion depends on success". Two students also felt that using Okapi avoided "hassling librarians" (70, 79). One student (2) gave her/his vision of subject searching: s/he said it is trying to find subjects which "fit" with "librarian-oriented" ways of describing subject matter. 8.3.3 Questions about their search

The full results are in Appendix 6 (Part 3). The main points are: only 19 (27%) students said they wanted to look for subjects; the majority of them said they came with a course-related reading list given by a teacher, and so were looking for specific items. Thirteen people out of 70 said they didn't find what they wanted. Once their search was over, most students (42 out of 70) said they only wrote down class number(s). 8.4 Analysis of user sessions

The full results are in Appendix 7. There are some discrepancies between the number of searches input and the number of searches processed. This

8. Okapi in use is because some searches were interrupted by the user. 8.4.1 Session length, types of searches, search states

155

On average, sessions lasted for 10 minutes and there were four searches per session. There were more searches for specific items (62%) than for subjects. This is not surprising because most students came with a reading list (Section 8.3.3). Out of 96 sessions, 62 contained only one ''search state", i.e. the user stayed in the same ''state" (started with, for example, an author search and did not change to any other type of search for that session). The most frequent sequences of search states are included in Appendix 7. This is not very enlightening; it would have been more interesting, for example, to look for the occurrence of pairs of search types in a session: for example the proportion of sessions which started with a specific item search and contained one or more subject searches, and whether the topics were related. There was not enough time, and it is difficult to extract this kind of information. There are also some problems to solve first, such as the definition of a ''search statement": when a search statement is entered several times in the same session at different intervals, is it the same search? What if the user does it again 10 minutes later? Is it necessary to decide on a maximum possible interval? (in Appendix 7, all search statements were counted). 8.4.2 Search failures

Similar problems also occurred in counting and analysing the types of matches by type of search processed. It is useful to know that about half (90 out of 190) the title/author acronym searches failed; but it would have been more useful to correlate these failures with, for example, spelling or input mistakes (Section 8.4.3). It would also have been interesting to know whether trying title only and author only afterwards was any use, and which was more useful, i.e. testing the "search trees" (Sections 6.8 and 7.4.3) to see whether the "paths" chosen are the most efficient ones. This would have been complicated to work out and very lengthy to record. It would be interesting to compare the failures found in the log with the user's answer to "did you find what you wanted". It would be

156

8. Okapi in use

interesting, too, to redo the searches done by the 70 students interviewed, look at the records to try to evaluate their relevance, and compare this with the students' answers. This is the advantage of being able to relate a piece of log with a particular interviewee (Section 8.2). The general conclusion is that there were many search failures, whatever the reasons, and that users did not complain very much about this, in particular about specific item search failures. 8.4.3 Use of keys, spelling and input mistakes

The analysis of these 96 sessions also confirms that the Black key was misunderstood. It was used wrongly in 77 cases out of 125. About 10% of the search statements contained spelling mistakes, but no attempt was made to correlate these with the length of search statements. A major result is that there are more problems with author input (18 % of author searches) than was signalled by the users themselves (see above "comments specific to Okapi" in Section 8.3.2); this could also be an important reason for specific item search failures. It is worth emphasising this point, especially since the team was particularly aware of this problem and worked for a long time on the design of the author input procedures. This is a complex problem. In spite of prolonged efforts by the team, this has not been solved satisfactorily. 8.5 Logging results

The full results are in Appendix 8. The automatic analysis of 3912 logged searches (corresponding to 28 working days) gives a proportion of subject searches (43% compared to 38%)) only slightly different from that found in the 96 analysed user sessions (which therefore can be said to be a fairly representative sample). The number of subject searches is not much lower than the number of specific item searches, which the library staff were interested to discover. The percentage of time spent doing each type of search (56% and 44% ) is almost identical to the percentage of searches done (57% and 43%). It would be interesting to see whether this changes with time, i.e. whether students venture more into subject searching, especially since many users think Okapi is more useful for subject searching (Section 8.3.2). Most specific item searches are author AND title searches. Note that the Okapi specific item input screen (Fig. 7.8) probably encourages the entry of

8. Okapi in use

157

both author and title. T h i s is one of the ways in which Okapi differs from most other OPACs.
T H E ANALYSIS OF SUBJECT SEARCHES

T h e average n u m b e r of words (excluding stop words) per subject search expression is just over two (calculated from 1652 subject search expressions). Not all multi-term search statements were A N D e d . For example, on 603 two-word subject search expressions, only 502 were A N D e d . T h e r e are two reasons for this difference: occasionally the user interrupted the search, but mostly at least one of the two terms had no postings (either because it was misspelt, or there was nothing in the index). This obviously diminishes as the n u m b e r of terms in the search expression increases: if there are more terms, even if one of t h e m does not have any postings, the system still A N D s the others. Obviously more A N D s fail as the n u m b e r of terms increases. T h e average n u m b e r of hits even for two-word search statements (30) is not very high. But it is only an average. T h e table in Appendix 8 shows the n u m b e r of h y p e r - O R s for each type of subject search, but not the n u m b e r of records looked at by the user. T h e results presented here are brief and sometimes only of limited interest, because of their incompleteness. T h e y are only the tip of the iceberg. M u c h more can be extracted from the logs but more work needs to be done on how to analyse logs in general. References 1 P e n n i m a n W S and D o m i n i c k W D . Monitoring and evaluation of online information system usage. Information Processing and Management 16 ( 1 ) , 1980, p l 7 - 3 5 . F e r g u s o n D K and others. T h e C L R public online catalog survey: an overview. Information Technology and Libraries 1 ( 2 ) , 1982, p8497. K a s k e N K and S a n d e r s N P . A comprehensive study of online public access catalogs: an overview and application of findings. Final report to the Council on Library Resources. Vol I I I . O C L C , 1983. S i e g e l E R et al. A comparative evaluation of the technical performance and user acceptance of two prototype online catalog systems. Information Technology and Libraries 3 ( 1 ) , 1984, p35-46. M o o r e C W. User reactions to online catalogs: an exploratory study. College and Research Libraries 42 ( 4 ) , 1981, p295-302.

2

3

4

5

158 6

8. Okapi in use

Norden D J and Lawrence G H. Public terminal use in an online catalog: some preliminary results. College and Research Libraries 42 (4), 1981, p308-316. 7 Borgman C L. End user behavior on the Ohio State University Libraries' online catalog: a computer monitoring survey. OCLC, Office of Research, 1983. 8 Pease S and Gouke M N. Patterns of use in an online catalog and a card catalog. College and Research Libraries 43 (4), 1982, p279-291. 9 Markey K. Subject searching in library catalogs: before and after the introduction of online catalogs. OCLC, 1984. 10 Cochrane P A and Markey K. Catalog use studies - since the introduction of online interactive catalogs: impact on design for subject access. LISR, 1983, p337-363. 11 Tolle J E et al. Current utilization of online catalogs: transaction log analysis. Final report to the Council on Library Resources. Vol I. OCLC, 1983. 12 Horowitz G L and Bleich H L. Paperchase: a computer program to search the medical literature. In: Proceedings of the 6th Annual Symposium on Computer Applications in Medical Care, New York, 30 October - 2 November 1982. Edited by B I Blum. IEEE, 1982, pl045-1051.