Chapter 5 Results

5.1

Experiment 1—holistic approach to searching in the library

The searching activity of library users was observed in its entirety in two d a t a gathering exercises: in the first the searches were initiated at the online catalogue and in the second the searches were initiated either at the catalogue or at the shelves. Although microfiche catalogues were still available in parallel with the online catalogue, none of the subjects used them. The OPAC1 sample was collected in the third month after the installation of the online catalogue in the middle of the summer term, whereas the second sample was collected in the middle of the first term five months later. The great majority of searchers had used the OPAC fewer than five times and a few between five and ten times in the first sample and most between five and ten times in the second sample. All had a very positive attitude towards the online catalogue with most volunteering how much they preferred it to the microfiche catalogue. The observation of the entire search process revealed that users looked for book materials in a variety of ways. They differed not only in the use or non-use of the online catalogue, but also in the number of steps they followed and the sequence or combination of those steps. T h e analysis of the trials concentrated on: • the identification of types of searches and their relation to search strategy, i.e. which search option (subject headings, keyword, author, title) searchers used to access the catalogue for what type of search • the search formulation in terms of expressed topic in relation to the initial access to the catalogue • the search outcome, i.e. success and failure both at the catalogue and at the shelves as well as the information extracted from the catalogue, (eg. specific reference or a class number(s) only).
35

5.1.1

Searching activity

Table 5.1 categorizes the different approaches in the use of the online catalogue and the shelves that were observed, and Tables 5.2 and 5.3 map out the initial approach and progress of all the searches in the 0PAC1 and OPAC2 samples respectively.

Table 5.1: Steps in searching the online catalogue and the shelves, grouped by category Step AA B C DD Description Online catalogue, author/title specific item search Shelves, specific item search Shelves, subject search Online catalogue, subject search

Table 5.2: Analysis of searches in the online catalogue followed at the shelves, showing the order of steps—oPACI SAMPLE. 1 Steps used in search, in o r d e r AA fAA AA tAA fAA tAA AA fAA tAA fAA No . of searches 24 3 20 7 4 3 1 1 1 1 4 3 8 2 11 2 3 1 1 100

B B

C C DD DD DD DD DD JDD DD DD JDD DD DD DD DD DD AA AA AA AA

DD

B B B

C C

B B B B

AA B AA DD AA B Total number of searches

c c c c c

B

fSearches initiated as specific item searches and then developing into subject searches. ^Apparent subject searches which were specific item searches.

36

Table 5.3: Analysis of searches in the online catalogue and at the shelves, showing the order of steps—OPAC2 SAMPLE. For f and } see table 5.2.

| Steps; AA AA tAA tAA tAA tAA tAA AA tAA tAA tAA

used in search, inL orde :r B B B B

No». of searches 12 12

C C C

B AA DD DD DD DD AA AA AA AA DD AA DD B B B B B

C C

c

C C 11

B B

tB tB

c c c c c c c c

AA 19 AA DD DD DD DD DD DD fDD DD DD DD JDD DD DD B B B B C

AA

C C

5 C 4 3 C 1 AA B C 1 AA DD AA C 1 AA Total number of searches 104

B B

5.1.2

Defining subject searches

T h e true identity of the different types of searches was not always evident at the outset of a search. To establish the type of search, the initial approach through to the final outcome had to be taken into account. It was found that out of a total of 128 cases initiated as specific item searches, (Tables 5.2, 5.3, initial categories AA or B), 43 (34%) developed into subject searches (including any subsequent category C, DD). Since these cases subsequently contained subject search characteristics, they were regarded as subject searches. Cases thus classified as subject searches are marked with a dagger in Tables 5.2, 5.3. For example, after initiating searches for known items, searchers may have: • looked for alternatives at the catalogue or at the shelves if they failed to find the intended item; • having found the intended item, looked for additional works either in the catalogue or at the shelves; • intentionally used a known item as a means of locating a relevant area at the shelves to carry out a subject search. Such searches, in spite of their initial objective or tactic, taken as a whole, have been deemed to be essentially subject searches and thus have been classed as such. The reverse, i.e. specific item searches disguised as subject searches occurred only eight times in the two samples (Category JDD). Table 5.4 compares the occurrence of hybrid and pure subject searches. Pure subject searches contain categories C, or DD (Table 5.2, 29 cases, Table 5.3, 39 cases). Those hybrid subject searches initiated as specific item searches at the online catalogue were further analysed and divided, as shown in Table 5.5, into those where the subject searching element occurred at the bibliographic tool (initial category f AA followed by category B only or DD, 9 cases Table 5.2, 6 cases Table 5.3, ) and those where the subject searching was introduced at the shelves, (initial category fAA followed by B and C or C, 11 cases Table 5.2, 9 cases Table 5.3). It appears that the subject searching element tends to be introduced at the shelves. The use and non-use of the online catalogue for the OPAC2 sample is shown in Table 5.6 which compares the subject and specific item searches and the type of search ctpproach adopted by the user in initiating the searches. The data relates to Table 5.3 as follows: • Subject searches at the online catalogue initial categories fAA and DD, 30 cases • Subject searches at the shelves initial categories fB and C, 32 cases • Specific item searches at the online catalogue initial categories A A and JDD, 27 cases • Specific item searches at the shelves: initial category B, 15 cases We find a predominance of subject searching, 60% (29% + 31%) and the use of the online catalogue stands at 55% (29% + 26%) 38

Table 5.4: Hybrid and pure subject searches Sample Hybrid subject searches 20 (41%) 23 (37%) 43 (39%) Pure subject searches 29 (59%) 39 (63%) 68 (61%) Total no of cases | 49 62 111

OPACl

OPAC2

Total

Table 5.5: Hybrid subject searches initiated at the online catalogue Sample Hybrid searches subject searching at the online catalogue 9 (45%) 6 (40%) 15 (43%) Hybrid searches subject searching at the shelves 11 (55%) 9 (60%) 20 (57%) Total no of cases 20 15 35

OPACl sample OPAC2 sample Total

Table 5.6: Subject and specific item searches initiated at the online catalogue and at the shelves Type of search OPAC2 sample no. of searches 30 32 27 15 104 %

Subject searches at a bibliographic tool Subject searches at the shelves Specific item searches at a bibliographic tool Specific item searches at the shelves Total no. of cases

29 31 26 14 100% |

39

5.1.3

Search formulation

Having identified subject searches in the initial quantitative analysis of the data sets, the next stage was to carry out a qualitative analysis on the search formulation of those cases. For the online catalogue, all searches which included a subject searching element at the catalogue in both samples, O P A C l (Category DD, 36 cases, Table 5.2), and OPAC2 (Category DD, 26 cases, Table 5.3), were combined into one data set totalling 62 cases. T h e analysis of search formulations at the online catalogue examined how a user's query or expressed topic is related to the initial access to the bibliographic tool, i.e. whether or not it is modified and if so how it is modified, whether it is broadened or narrowed. This corresponds to Taylor's (1968) formalized need (Q3) and the compromised need (Q4) and is referred here as 'the initial search formulation'. Unlike the case of a reference enquiry, the query here is presented directly to the system without the benefit of an intermediary to allow for negotiation. Each transition was classified as exact, narrow to broad, or broad to narrow, according to the contextual relationship between the terms of the expressed topic, and the initial access point, i.e. what the user typed in first. In some cases the contextual relationships could have been described by categories other than exact, broad, and narrow, however it was found that the hierarchical interpretation suited the majority of cases. Subject interests were expressed in a very concise manner, on average two terms only. Access to the online catalogue revealed an interplay of two approaches: contextual and matching. In the former searchers altered the expressed topic in order to find relevant terms in the tool and in the latter they attempted to match the expressed topic directly with the tool. Initial access terms which were partially different from the expressed topic may have encompassed either or both approaches, straight matching a n d / o r contextual. It was not possible to ascertain in which circumstances a user would adopt a matching approach as opposed to a contextual one or vice-versa. For example in the case of 'short term and long term memory' being given as an expressed topic, it is unclear whether the choice of the term 'memory' only as an access term instead of 'short term memory' was an attempt to find the most appropriate term to match the language of the index or whether there was a attempt to place the query in a broader context to include both specific aspects, i.e. short term and long term memory. Table 5.7 shows the contextual relationship between the expressed topic and the terms chosen for the initial access to the online catalogue. Having chosen their initial access points, 12 out of 62 cases (19%) didn't succeed in their initial access, whilst another 15 cases partially matched their initial access points, either through the system displaying a closest match or the searcher browsing subject headings. Table 5.8 indicates the systems options users selected in carrying out subject searches on the online catalogue. These include the first as well as subsequent choices. The order of the options presented in the opening menu may have influenced their approach.

40

Table 5.7: Expressed topic and initial access to the online catalogue T y p e of approach Exact Narrow to broad Broad to narrow Total No. of cases 36 17 9 62

~%]
58 27

15 1
100 j

Table 5.8: Search options in the online catalogue Option Searched by subject headings Searched by keyword access Searched by both Total No. of cases 34 19 9 62
%

55 31 14 100

Table 5.9: Orientation tactics in the online catalogue Tactic | Consulted initial access point only | Browsed subject headings Browsed subject headings and generated new terms Browsed subject headings, generated new 1 search terms and changed search strategy Browsed subject headings and changed search strategy Generated new search terms only Generated new search terms and changed search strategy Changed search strategy only Total No. of cases 11 12 11 6 3 9 9 1 62
%

18 19 19 10 5 14 14 2

100 1

41

Once searchers had accessed the online catalogue, they then followed various tactics as shown in Table 5.9. A change in strategy here refers to switching from a strict subject approach to include a specific item approach or vice versa, i.e. a hybrid search. Searchers thus tended to refine their searches by browsing subject headings (52%), generating new terms (56%) or changing search strategy (31%).

5.1.4

Search outcomes

A third aspect of the analysis was concerned with determining success/failure both at the catalogue and at the shelves for all subject searches undertaken in both environments. They included the following 83 cases: • Subject searches initiated at the online catalogue, (initial category DD OPAC1 sample Table 5.2, 29 cases, and OPAC2 sample Table 5.3, 15 cases) • Subject searches initiated as specific item searches at the catalogue but developing into subject searches either at the catalogue or at the shelves, (initial category fAA O P A C l sample Table 5.2, 20 cases, and OPAC2 sample Table 5.3, 15 cases) • Subject searches which were initiated at the shelves but were continued at the online catalogue, (initial category C + DD, OPAC2 sample Table 5.3, 4 cases) Comparisons were drawn to find out: a) Success/failure rates at the online catalogue b) Subsequent success/failure rates at the shelves for cases followed up from the online catalogue c) Final success/failure rates of cases initiated at the online catalogue The results of the consultations of the online catalogue are given in Table 5.10 . Success and failure can be defined in different ways. In particular cases with references found but not available for loan could be considered as failed searches. Others however chose a class number as an alternative and were thus considered as successful searches. In Table 5.11 the final result at the shelves for subject searches initiated at the online catalogue are presented. Of the original 83 cases, 10 cases failed at the catalogue, 6 of which found references not available for loan and did not pursue their search further. T h e final success/failure rate at the online catalogue is 82%/18%. A high proportion of searches were satisfied with the titles they followed up from the online catalogue and did not search any further, (23%). The selection of titles found in the online catalogue also promoted the selection of other titles at the shelves, (30%). The retrieval of titles from the shelves via class numbers extracted from the bibliographic tool appears less effective For class number searches from the online catalogue the rate is 6 6 % / 3 3 % (10 and 5 cases respectively). 42

Table 5.10: Outcome of subject searches at the online catalogue Result Found reference(s) Found class number(s) Found reference(s) but not available for loan No references found Total No. of cases 58 15 6 4 83

%]
70 18 7 5 100

Table 5.11: Final result at the shelves for subject Searches initiated at the 1 Result Selected only title(s) I found at the catalogue Selected title(s) found at the catalogue &; others Did not select any title(s) found at the catalogue but selected others Did not select any title(s) at all Selected title(s) from a class number(s) Did not select any title(s) from a class number(s) Total online catalogue No. of cases 17 22 11 % 23 30 15

8 10 5

11 14 7

73

100 j

Table 5.12: Success/failure rates at the online catalogue and at the shelves Success no. of cases 73 60 % 88 82 Failure no. of cases 10 13 % 12 18 Total 83 73

Online catalogue Online catalogue/ shelves

43

Table 5.12 compares success/failure rates for searches both at the online catalogue and at the shelves. The success rate was well maintained at the shelves, dropping only 6%. In the final outcome however searchers selected on average only 1.9 items from the shelves.

44

5.2

Experiment 2—subject searching strategies

The holistic approach of Experiment 1 showed that 39% (Table 5.4) of subject searches were identified as hybrid searches initiated as specific item searches, 43% of which were undertaken at the catalogue and 57% developed into subject searches at the shelves (Table 5.5). The second experiment sought to further explore the holistic approach without following the searcher to the shelves. The aim was to attempt to elicit user's intention with respect to subject searching strategies. The objectives were twofold: • To ascertain whether users would reveal their intent to undertake a subject search via a specific item search if questioned either before accessing the catalogue, or after the catalogue consultation but prior to searching at the shelves. • To determine to what extent the screen logging facility represented an accurate record of the searching activity and confirmed the searcher's declared intention.

5.2.1

Methodology and validation of data gathering m e t h o d s

In order to address the research question pertaining to the identification of subject strategies, the following methods of eliciting information from catalogue users were tested: a) a self administered offline pre-search questionnaire (Appendix B l ) b) an online pre-search questionnaire (Appendix B2) c) a self administered offline post-search questionnaire (Appendix B3) d) an online post-search questionnaire (Appendix B4) e) a screen log of search sessions A basic multiple choice format was adopted. This was then adapted depending on the online or offline environment and at which stage the questionnaire would be presented to the searcher. Separate data sets were collected for each questionnaire and a screen log was recorded for each case. Searchers were given the opportunity to declare their strategy or intended strategy at two stages in each of the questionnaires. Firstly in a direct form at the outset when asked what they wanted and secondly indirectly following an anticipated or actual outcome, when asked how they would pursue their search as a result of the outcome. The different methods adopted for elicitating this verbal d a t a were compared and validated in a number of ways: • The verbal d a t a obtained from each type of questionnaire was compared for reliability with the non-verbal data provided by the screen logs. 45

• Pre-search and post-search questionnaires were compared to ascertain users' perception of the search before and after the event. • Online and offline questionnaires both before and after searches were compared to assess the influence of the different medium.

5.2,2

Results of Experiment 2

Three types of searches were identified and classified as follows: a) Pure subject searches were subject searches declared in the questionnaire and confirmed by the log. b) Specific item searches were searches declared as such in the questionnaire and confirmed by the log. c) A hybrid search was identified if the search met one of three conditions. Firstly if the searcher declared in the questionnaire that the search was or would be a combination of a specific item and subject search. Secondly a search could be identified as a hybrid by default if the searcher was unable to declare whether the search was one of the two types. Thirdly searches which were declared in the questionnaire as either a specific item search or a subject search but where the log indicated otherwise, were also classed as hybrid searches. We thus differentiate between a declared and undeclared hybrid search (searches assumed hybrid by default are included with those declared hybrid. In the former case, the searcher declares in the questionnaire the intention to adopt or the fact of having adopted both a specific item and subject approach in pursuing a search. In the latter case, there is indication in the questionnaire that the searcher does not intend to undertake, or has not undertaken, a hybrid search but the log reveals that a hybrid search has been undertaken. Alternatively, an undeclared hybrid search could also be a search identified either as a specific item or subject search in the questionnaire but which turns out to be the opposite in the log. Table 5.13 presents the different types of searches for each of the samples for which a different d a t a gathering method was adopted. The combination of both subject searching strategies, i.e. pure and hybrid subject searches, reveals an overall dominance of 72%. Hybrid searches account for the highest proportion (45%) of all types of searches. P u r e subject searches ranged between 16% and 35% of the different types of searches and 37% of all subject searches overall, whereas hybrid searches ranged between 31.5% and 63% of the total number of subject searches. In Table 5.14 the 82 hybrid searches are divided between those which were declared in the questionnaires and those which were not. Each data set is further divided to indicate the identity of each search if the information derived solely from the log is taken into account. In the case of the undeclared hybrid searches, both the identity derived from the questionnaire and from the logs are given, thus indicating the discrepancy between the verbal and non verbal data which led to the individual search being classified as a hybrid search.
46

Table 5.13: Experiment 2 : Types of Searches 1 T y p e of Questionnaire Specific item 19 (38%) 11 (23%) 8 (16%) 13 (37%) 51 (28%) T y p e of S e a r c h Hybrid Pure Subject 8 (16%) 23 (46%) 12 (26%) 24 (51%) 17 (35%) 24 (49%) 11 (31.5%) 11 (31.5%) 48 (27%) 82 (45%) Total No. of cases 50 47 49 35 181

Pre-search offline Pre-search online Post-search offline Post-search online Total no. of cases

Table 5.14: Experiment 2: Declared and Undeclared Hybrid Searches T y p e of Q u e s t i o n n a i r e Pre-search offline (54%) Pre-search online 12 Declared hybrid searches 16 logs Undeclared hybrid searches 7 4 SI 3 SU 12 4 SI 8SU 6 1 SI 5SU 5 2 SI 3SU 30 (37%) logs

Post-search offline (46%) Post-search online T o t a l n o . of c a s e s

18

6 52 (63%)

9 SI 2SU 5 H 7 SI 4SU 1 H 13 SI 2SU 3H 4 SI 2 H

3 SU 1 H 1 SI 2SU 2H 6 SI 1 SU 4 SI 1 H 2SU 2 SI

47

It would appear that irrespective of the timing of the questionnaire before or after the catalogue consultation, or the medium, searchers declared their intention to undertake a hybrid search as a subject searching strategy, 54% for the presearch questionnaires and 46% for the post-search questionnaires. A closer analysis of the logs of the declared hybrid searches, indicates that 11 out of the 52 carried out a hybrid search at the catalogue, but 33 carried out specific item searches with the declared intention in the questionnaire of pursuing a subject search at the shelves. The remaining 8 cases who carried out apparently pure subject searches at the catalogue would appear to have used the subject searching option as a means of searching for both known items and unknown items on the same topic. In spite of the use of the catalogue, users thus appear to be clearly stating their intention to adopt a hybrid subject searching strategy and to pursue subject searching at the shelves. Undeclared hybrid searches which account for 37% of hybrid searches reveal a similarity of intent with 13 out of 30 logs indicating the use of specific item searches as a means or strategy of undertaking a subject search with 6 other cases actually carrying out hybrid searches at the catalogue. The remaining 11 cases however show the reverse situation, that is a subject or hybrid search undertaken as a means of carrying out a specific item search. In comparing the verbal data elicited from the user through the questionnaires and the non-verbal data derived from the logs, it is evident that the logs considered independently are not a reliable source for identifying types of searches. Out of 181 searches, 110 (61% i.e. 51 specific item, 48 pure subject, and 11 hybrid searches) have logs which match the information provided by the corresponding questionnaires. In 71 cases (39%) however, the logs did not confirm what the searchers declared in the questionnaire.

48

5.3

Experiment 3—search articulation & formulation

Experiment 1 revealed that searchers on average expressed their search topics with only two terms and also tended to use those same terms to access the online catalogue (Table 5.7). If Taylor's categories of information need were applied, it would appear that there is very little difference, if any, between the formalised need (Q3) and the compromised need (Q4), (Taylor, 1968). One explanation could be that there is already a strong element of compromise at the formalised stage. In view of the close relationship between expressed topic and initial search formulation, the aim of the third experiment was to attempt to establish a clearer distinction between the way searchers expressed their topics in their own terms and the way they formulated their search in accessing the catalogue. The objectives were as follows: • To encourage searchers to articulate their search more freely. • To establish how topics expressed more freely would relate to how a searcher formulates a search in the online catalogue. • To determine whether searchers could articulate their search differently before or after searching the catalogue.

5.3.1

Methodology and analysis

D a t a was collected from four different groups of searchers using a version of a common questionnaire adapted to suit the environment and timing of the particular questionnaire. As with the previous experiment, each search was logged. The test instruments for Experiment 3 included the following questionnaires: • self-administered offline pre-search (Appendix CI) • online pre-search (Appendix C2) • self-administered offline post-search (Appendix C3) • online post-search (Appendix C4) The questionnaires consisted of a single question inviting searchers to declare what they were looking for and to describe it as fully as possible. The example provided suggested implicitly that they could express what they wanted in terms of a specific item or a topic. In the initial analysis the questionnaires and the logs were used to identify the type of searches being undertaken as described in the previous experiment, (Section 5.2.2). The pure subject searches and hybrid searches were then further analysed: 49

a) To determine the relationship between the expressed topic as articulated in the questionnaire by the searcher and the search terms used to formulate the search in accesssing the catalogue. b) To determine the influence on this relationship of the method of eliciting the articulation.

5.3.2

Results of Experiment 3

Table 5.15 shows the distribution of the different types of searches across the different d a t a sets. In comparison to Experiment 2 (Table 5.13) the low occurrence of hybrid searches 16% as opposed to 45% could possibly relate to two factors. Firstly, the holistic approach adopted in Experiment 2 took into account searchers' intention with regard to pursuing their search after the catalogue consultation, i.e. at the shelves which Experiment 3 did not. Secondly, the first multiple choice question in Experiment 2 included the possibility of a hybrid search whereas in Experiment 3 this approach was not suggested. In the case of items from a reading list, searches were classed as they were perceived by the searcher. For example if searchers declared that they were looking for known books a n d / o r listed them, the search was classed as a specific item search. If however, they went further and qualified or provided a subject description, those searches were classed as hybrid searches. Although some searchers did declare that they would undertake a hybrid search, it is likely that others remained undeclared and consequently undetected. On the other hand, the number of pure subject searches in both experiments were comparable (27% and 28% respectively). Table 5.15: Experiment 3: Types of Searches T y p e of Q u e s t i o n n a i r e Specific item 22 (65%) 51 (53%) 28 (61%) 24 (50%) 125 (56%) T y p e of Pure Subject 8 (23%) 32 (33%) 7 (15%) 16 (33%) 63 (28%) Search Hybrid 4 14 11 8 37 (12%) (14%) (24%) (17%) (16%) Total no. of cases 34 97 46 48 225

Pre-search offline Pre-search online Post-search offline Post-search online Total no. of cases

Each of the search statements articulated in the questionnaires was compared with the different terms used by the searcher to access the catalogue. These included initial as well as subsequent access terms. Each case was then classed according to what extent the written search statements were reflected in the overall search formulations. T h e cases were found to fall into three categories: a) The search articulations which were accurately or exactly represented in the search formulations.
50

b) Those cases who articulated their search in a more specific or qualified manner than that revealed by the terms used to search the catalogue. c) The searches whose statements were couched more broadly than the actual search formulations.

Table 5.16: Experiment 3—Search Articulations compared with Formulations T y p e of Q u e s t i o n n a i r e Exact Pre-search offline 50% Pre-search online Post-search offline 55% Post-search online Total n o . of c a s e s 13 (54%) 52 (52%) 24 (52%) 10 (56%) 26% 5 (21%) 31 (31%) 5 (42%) 34% 15 (33%) 6 (33%) 19% 6 (25%) 17 (17%) 24 100 T y p e of Articulation Narrow Broad 5 (42%) 16% 7 (15%) 2 (11%) 46 18 2 (16%) Total no. of Cases 12

Although 52% of search articulations did match the search formulations, as seen in Table 5.16, 48% did not. The proportion of exact, narrow and broad search articulations for the combined types of pre-search and post-search questionnaires was of the same order with the exact articulations having the highest occurrence followed by the narrow and broad articulations. It would appear that there is nothing to distinguish the pre-search or post-search articulations and that the catalogue consultations had no apparent effect on these articulations. The average number of significant terms which were included in the search articulations are presented in Table 5.17. Although searchers were encouraged to be discursive the overall average for all types of articulations and all types of questionnaires was 2.9 terms compared with 2 terms in Experiment 1. Narrow articulations tended to be less concise with 4.3 terms on average overall followed by broad articulations with 2.5 terms and exact ones at 2.3 terms. The online statements tended to be slightly more concise than the written offline versions but not significantly so.

51

Table 5.17: 15 Experiment 3—Number of terms in search articulations Average no. Type of Questionnaire Type of articulation of Terms 3 Exact Pre-search offline Narrow 3.8 1.5 Broad Overall 3 Exact 2.1 Pre-search online Narrow 3 2.6 Broad Overall 2.4 Exact 2.3 Post-search offline Narrow 7.1 4 Broad Overall 4.1 Exact 2.3 Post-search online Narrow 5.3 Broad 2.2 Overall 3

52

5.4

Experiment naires

4—Olive

interactive

question-

In Experiment 4 the aim was to explore the use of interactive online questionnaires as a more direct means of eliciting information from the user as opposed to the pre-search and post- search, online and offline, paired tests carried out in Experiments 2 and 3. The objectives were: • To determine convenient and appropriate points whereby a search could be intercepted. • To assess the reliability of the method in relation to other methods being tested. • To explore to what extent the method could provide more information on the search process.

5.4.1

Methodology

Two questionnaires were tested (Appendix D l , D2) to address research questions comparable to those of Experiments 2 and 3 relating to subject searching strategies and search articulation respectively. In all cases the 'Olive' interception was triggered by a restart after the display of a bibliographic record containing circulation data (i.e. copy availability screen, Section 4.2.2). Different conditions were set for each of the questionnaires (Section 4.2.3) and the sequence of the questions also depended on replies given to previous questions. Experiment 4A in essence replicated Experiment 2 as described in Section 5.2.1. The main difference was that searchers were asked what further action they intended to pursue at the catalogue a n d / o r at the shelves with regard to particular record(s) they had viewed. In addition there was the possibility of questions being repeated several times during a particular search session, ie. each time the user pressed the restart key following the viewing of a 'copy availability' screen in the previous search sequence. Experiment 4B differed slightly from Experiment 3 (Section 5.3.1) in that searchers were preselected in so far as those choosing a subject search option from the opening menu, met part of the condition for the interception. Searchers were not only asked to express their search requirements as fully as possible in a free text question but were also asked to specify whether the item(s) displayed, satisfied or partly satisfied their requirement. If an outcome partly satisfied their requirement they were asked to qualify whether the item(s) were related, narrower, or broader.

5.4.2

Results of Experiments 4A and 4B

Due to the technical constraints and complexities as discussed in Section 4.4 which led to the belated implementation of Olive, the small samples collected can only be considered as an indication of the potential of interactive questionnaires. The data on the subject searching strategies collected by this method as shown in Table 5.18, would seem to be comparable to that collected in Experiment 2, with 53

hybrid searches being declared through the questionnaire and not necessarily being evident in the log. Table 5.18: Olive—Experiment 4A Type of Searches Type of Search Specific item Pure Subject

No. of C.ases
15 2 7 Declared Undeclared 6 1 SU

logs

Hybrid

SI H SI

3 3 1

Total no. of cases

24

In Table 5.19 search articulations are classed according to how they compared with the search terms chosen by the searcher to access the catalogue. Three cases whose expressed topics corresponded exactly with search terms found items which met their requirements, whereas two cases found items which were related. Narrow search articulations would appear to be less successful. Table 5.19: Olive—Experiment 4B Search Articulations compared with search formulations and search outcomes No. of Search Type of search Cases Outcome articulation 5 3 exact Exact 2 related 4 1 exact Narrow 2 related 1 narrow 9 Total no. of cases Exact search articulations averaged 2.6 terms whereas narrow articulations averaged 4.2 terms.

54

5.5

Experiment 5: search replays

Although the screen logging facility provided an unobtrusive method of recording a search in its entirety, the recording of the searching activity at the terminal was predominantly from the system's perspective. The replay facility was developed in conjunction with the logging facility as a means of considering the retrieval task from the user's perspective without the risk of influencing or interferring with the actual search process. The method was thus tested: • To assess users' perception of the search and retrieval task. • To allow users to identify decision points and to offer explanations for decisions taken including relevance judgements. • To indicate users' expectations and satisfaction of search outcomes.

5.5.1

Methodology

A small sample of nine searchers who were observed to be undertaking a subject search on the online catalogue whilst participating in Experiment 3, were invited to have their search replayed immediately after they had completed their search and declared their topic in the post-search offline questionnaire. They were asked to 'talk through' their search and provide explanations on decisions and comments on results as it was being replayed to them. This was done in real time providing an exact replication or simulation of their search. Although a pause facility was available this was not used. During the course of the replay the experimenter on occasion prompted searchers to indicate what they were looking for and to explain choices made. For the most part however searchers were most forthcoming in providing explanations. A summary of users' comments were recorded in writing by the experimenter together with some additional notes.

5.5.2

Results of Experiment 5

Due to the exploratory nature of the methodology and limited number of cases, the somewhat anecdotal information gathered in this experiment cannot be quantified. Nevertheless the small sample did produce some useful qualitative data from which indicative characteristics of subject searching behaviour could be derived. These fall under several general headings. Information
4

neecT

Although all the searchers clearly expressed a formalized need prior to the replay, in 5 out of 9 cases it emerged in the course of the replay that what they were seeking was far more nebulous than what they had indicated either in the search terms or expressed topic. Some of the searchers declared that they were searching the catalogue in order to define what they were looking for. For example, the searcher who expressed his
55

formalized need as 'corporate finance', searched under ther terms 'funding' followed by 'banking'. Although he found a title which included the terms 'corporate funding', which he considered relevant, he also explained that he wanted different aspects of corporate funding but he wasn't sure what these might be. By using the search term 'banking' he looked for clues in the titles and found that terms like 'equity', 'stock exchange' and 'borrowed funding' might be of use. He did not however use these terms to further his search at the catalogue, but noted the items as possible items to consult on the shelves. It would seem that Taylor's categories of information need in such circumstances, are perhaps too rigid and that Belkin's model of anomalous state of knowledge (ASK) may be more appropriate. (Taylor, 1968; Belkin et al, 1982). There is further evidence to support this. For instance two searchers also divulged that they would adapt their requirement according to what they found in the catalogue. T h a t is, they would alter or adjust their project topic according to, or as a result of the documents they could find in the catalogue as in the following example. A searcher expressed her requirement as looking for: information on international law of migration, the United Nations and 'race' or 'refugees', immigration: Britain, restrictions and practices, stateless persons and their movements. She commented after searching under the terms 'United Nations', 'race', 'international migration', 'refugees', 'race and colonisation' and 'immigration': I have looked for things before, I am really looking for the theory behind the policies but all of these (references) tend to deal with the law so I'll have to tackle it that way. Other searchers also indicated that the catalogue consultation did not represent their first attempt at finding information on their topic. One case expressed that it was her third attempt at looking for material on a topic, having consulted the catalogue and shelves twice previously. Searchers clearly considered the catalogue consultation as part of an ongoing process and not as a single independent event. Matching the s y s t e m language In eight of the nine cases, users' formalised need was declared as a qualified statement from which terms were selected to access the catalogue. Users selected those which they thought would most likely succeed in providing a posting even if those terms were broader than their requirement. Searchers thus consciously took a broader strategy. This proved to be a successful approach in so far as for all cases the selected search terms resulted in a posting. Nevertheless one searcher objected that the option to search by subject headings had been withdrawn by the library. She had found it a useful source for suggesting possible search terms. "Better than nothing" she commented.
56

Refining s e a r c h e s The broad approach to search formulation led to high postings rather than the reverse. Users responded to high postings in one of two ways. They restarted their search immediately by using new search terms to limit the search, or browsed through some the references until they found something of interest. Postings considered to be high by the searchers ranged from 84 to 1000, although in these particular two cases the user browsed through several screens. On the other hand postings of 93 and 241 prompted an immediate restart of the search. The different response to high postings did not apparently bear any relationship to the type of search i.e. whether or not the searcher had a clear idea in mind or was experimenting. For example one case searching for 'health statistics', entered 'health' as a search term and browsed 14 out of 159 screens until he found a title containing both terms 'health' and 'statistics'. On the other hand another case with the expressed topic 'history of drama', entered the term ' d r a m a ' which resulted in a posting of 241 references. The searcher restarted and entered the terms 'history of d r a m a ' which resulted in 21 references. It then emerged during the replay in browsing those references, that the searcher was unclear about what he was looking for and that perhaps something on English theatre and drama might be useful. Refining searches either by browsing references or by using new search terms appears to be based on ad hoc decisions. Although in the example of health statistics a Boolean operation may have been more fruitful, in the case of the history of drama it is unclear what help could have been provided with the search mechanisms available in present day systems. S e a r c h o u t c o m e s and satisfaction Searchers noted both specific references and / or class numbers as a result of the catalogue consultation. Four out of the nine cases recorded class numbers only to pursue their search at the shelves. Three out of those four had not viewed any individual records but only browsed screens of brief references. The remaining five cases noted individual references which they would look at in the first instance at the shelves, but also indicated that they would need to look for other items at the shelves, for they did not consider the items retrieved from the catalogue alone could adequately meet their requirements. It could be said that for all of the cases the catalogue consultation was regarded by the users as tentative and provided limited satisfaction. Judgements of success or failure at t h a t point were considered premature.

57