VI-1

VI.

A Comparison Between Manual and Automatic Indexing Methods G. Salton and D. K, Williamson

Abstract The effectiveness of conventional document indexing is compared with that achievable by fully-automatic text processing methods. Evaluation results are given for a comparison between the MEDLARS search system used at the National Library of Medicine, and the experimental SMART system, and conclusions are reached concerning the design of future automatic information systems.

1.

Introduction The design and operations of large-scale information systems has

become of concern to an ever-increasing segment of the scientific and professional world. Furthermore, as the amount and complexity of the

available information has continued to grow, the use of mechanized or partly mechanized procedures for various information storage and retrieval tasks has also become more widespread. As a result, a number of large

information systems are now in operation in which at least the search operations — that is, the comparison of incoming search requests with stored information — is carried out automatically. Typical examples in

the United States are the NASA Scientific and Technical Information Facility, and the MEDLARS system at the National Library of Medicine.

VI-2

While these operational information systems are thus able rapidly to search vast storage files, often containing many hundreds of thousands of items, most of the operations other than the search itself are performed manually with the help of human experts. In particular, all the

content analysis and indexing operations, leading to the assignment of suitably chosen combinations of index terms to the stored documents and to incoming search requests are normally performed by specialists who know the given subject area, as well as the performance characteristics of the retrieval environment within which they operate. Many of the information systems which base their operations on manual indexing but largely automatic search methods are quite successful in isolating, from the large mass of largely irrelevant stored material, many of the items which prove pertinent to the users' information needs. Nevertheless, the feeling that manual systems and procedures should be replaced by suitably chosen automatic methods has continued to grow, and a number of fully-automatic information storage and retrieval systems have been designed and put into operation, at least on an experimental basis. The SMART system represents one such effort to replace the intellectual indexing by sophisticated automatic text analysis procedures, and thereby to produce a retrieval environment in which all document and query handling procedures are performed automatically [1,2,3]. In the next few paragraphs, some of the evaluation measures that have been widely used to determine the effectiveness of information systems are introduced, and typical evaluation results obtained with the SMART system are given. Thereafter, the design of the SMART-MEDLARS test is

VI-3

examined, and evaluation results are given for the comparison between SMART and MEDLARS searches, using a variety of different analysis and search methods. Suggestions are made for improving the performance of presently

operating information systems, and for the design of future automatic retrieval services.

2.

The Evaluation of Information Systems Many different criteria may suggest themselves for measuring the

performance of an information system.

In the evaluation work carried out

with the SMART system, the effectiveness of an information system is assummed to depend on its ability to satisfy the users1 information needs by retrieving wanted material, while rejecting unwanted items. Two

measures have been widely used for this purpose, known as recall and precision, and representing respectively the proportion of relevant material actually retrieved, and the proportion of retrieved material actually relevant [4,5]. (Ideally, all relevant items should be retrieved, while

at the same time, all nonrelevant items should be rejected, as reflected by perfect recall and precision values equal to 1 ) . It should be noted that both the recall and precision figures achievable by a given system are adjustable, in the sense that a relaxation of the search conditions often leads to high recall, while a tightening of the search criteria leads to high precision. Unhappily, experience has

shown that on the average, recall and precision tend to vary inversely since the retrieval of more relevant items normally also leads to the retrieval

VI-4

of more irrelevant ones.

In practice, a compromise is usually made, and a

performance level is chosen such that much of the relevant material is retrieved, while the number of nonrelevant items which are also retrieved is kept within tolerable limits. In the SMART evaluation system, these various possible operating ranges are taken into account by computing for each search request, and for each processing method a variety of different statistics related to recall and precision. Specifically, four global statistics are generated, known

as rank recall, log precision, normalized recall, and normalized precision respectively, as well as ten local statistics, consisting of the standard precision at ten different recall levels. The global statistics are used to represent the overall performance of a given search, whereas the local statistics furnish individual recall-precision pairs for specific operating ranges of the system. Paired comparisons are normally presented, consisting

of the average performance over many search requests of two given search and retrieval systems [5]. One of the document collections used for evaluation purposes with the SMART system over the last few years is the set of 200 documents and 42 search requests in the field of aerodynamics used earlier as part of the well-known Aslib-Cranfield experiments [4]. This collection is attractive for test purposes since a number of actual user queries were available, as well as sets of relevance judgments obtained from the scientists constituting the user population. Furthermore, English abstracts were fur-

nished with each document, and it thus became possible to compare the effectiveness of the conventional retrieval operations based on a matching

VI-5

of the index term sets — manually assigned by trained indexers at Cranfield — with the performance of the fully-automatic language processing devices based on the manipulation of document abstracts, used by the SMART programs. Such a comparison could then produce evidence to indicate

whether document identifiers automatically generated by language analysis methods, such as suffix cut-off procedures, thesaurus look-up, phrase generation methods, statistical term associations, syntactic analysis, and others, would perform equally as well as manually assigned index terms. A typical comparison between the Cranfield indexing, and an automatic word stem matching process based on a matching of weighted word stems extracted from document abstracts and search requests, respectively, is shown in Fig. 1, averaged over the 42 Cranfield queries. The recallprecision graph of Fig. 1(a), and the corresponding tables of Fig. 1(b), indicate that the manual indexing is slightly superior to the simple automatic word-stem process. However, the statistical significance com-

putations, included in Fig. lCc), show that the differences in performance between the two systems are not significant. Specifically, each of the

values shown in Fig. 1(c) represents the probability — computed by using either a standard t-test, or a sign test — that if the performance of the two systems (manual indexing and automatic word-stem match) were, in fact, equally high, then a test value as large as the one actually observed would occur in practice [5]. A probability of 0.05 is usually taken as an upper bound in judging whether a deviation in test values is significant or not. The probability values included in Fig. 1(c) are seen

to be much higher than 0.05, and the assumption that the two systems are approximately comparable in effectiveness cannot safely be rejected.

VI-6

Prec islon
i

o—o Abstract, Stem (SMART) a—o Index, Stem (Cranfield)

1.0
0.8 0.6 o.>o^ 0.4 0.2 0^°-0^.n ideal system

-<S
^—r^—-=±—=^ 0.2 0.4 0.6 0.8
a)

0

H^~ Recal I 1.0

Recall-Precision Graph

SMART Word Stem R 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 RNK LOG NOR NOR P 0.8239 0.6518 0.5578 0.5093 0.4522 0.4143 0.3800 0.3431 0.3005 0.2551 0.2998 0.4655 0.8644 0.6704

Cranfield Indexing R 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 RNK LOG NOR NOR REC= PRE= REC= PRE= P 0.8045 0.6581 0.5908 0.5498 0.5171 0.4506 0.4035 0.3649 0.3233 0.2799 0.3122 0.4674 0.8897 0.6831

Evaluation Measures Precision at R = 0.1 R = 0.3 R = 0.5 R = 0.7 R - 0.9 RNK LOG NOR NOR REC PRE REC PRE

Probabilities t-Test Sign Test (Indexing over Abstract)

0.5151 0.4358 0.1163 0.3682 0.4044 0.6622 0.9341 0.1491 0.7268 0.0415

0.7011 0.7283 1.0000 0.8679 0.2559 0.0470 1.0000 0.1081 1.0000 0.0465

REC= PRE= REC= PRE=

Combined

b)

Recall-Precision Tables

c)

Significance Output

Recall-Precision Comparisons for Cranfield Indexing and SMART Word Stem Process (averages 42 queries, 200 document abstracts, cosine numeric) Fig. 1

VI-7

The results of the SMART-Cranfield comparisons seem to indicate that even simple automatic text analysis procedures do not necessarily produce retrieval results which are much inferior to those obtained in a system based on manual indexing. In fact, the benefits of the index

language control supplied in the conventional retrieval situation by the human indexers appears to be balanced by a deeper and more complex type of analysis available in the automatic environment, including selective term weighting and the use of relatively large sections of text to insure a high degree of indexing exhaustivity. While the results of the SMART-Cranfield test are in line with many other evaluation figures obtained by SMART with different document collections in other subject fields [5] f the test has nevertheless been criticized by some writers. In particular, it has been claimed that [6]:

a)

the use of the standard recall-precision measures is questionable, since other possible criteria (cost, waiting time, etc.) are disregarded;

b)

the relevance of a document with respect to a search query is not a stable criterion but varies with the user population, thus presumably producing different evaluation results for different sets of users;

c)

the experimental controls used to identify the Cranfield user population, the query set, and the sets of relevance judgments may have been deficient;

d)

the sources of variation affecting systems performance are not pinpointed, and no indication is given to permit a generalization of the test results to large, operational situations.

VI-8

Certain recent experiments appear to indicate that some of these objections may be groundless — for example, different user populations

seem to agree on the relative ordering of a set of documents in decreasing order of relevance with respect to a search request, thereby producing constant recall and precision values [7]. However, further comparisons between manual and automatic indexing systems are certainly of interest. The experiments carried out with a small subset of the MEDLARS collection were undertaken in an attempt to obtain further evidence in the ongoing comparison of conventional and automatic information systems.

3.

The Test Design

A)

The MEDLARS Evaluation Study

The SMART-MEDLARS experiments to be described are based on a small portion of a much larger systems evaluation study undertaken over the last few years within the National Library of Medicine [8,9]. In this

larger study, 302 search requests actually processed by MEDLARS were carefully chosen to reflect both a stratified sample of the MEDLARS user population, and a representative proportion of the subject fields covered by MEDLARS. For these 302 searches, the help of the users was enlisted in

order to obtain careful value judgments, made on a sample of the search output for each query. Specifically, a precision base (PB) was con-

structed by judging for relevance a sample of the documents retrieved by MEDLARS in response to each query; similarly, a recall base (RB) was obtained by taking documents from a variety of sources which were identified

VI-9

in advance as being relevant to the query*

The recall and precision

base documents were then used to compute for each search the recall and precision actually achieved during the search of the MEDLARS collection. While the design of the complete in-house MEDLARS test cannot be covered here, it is of interest to examine briefly some of the principal results: The overall average recall figure for MEDLARS was found to be

approximately 0.58, while the overall precision was 0.50, thus indicating that, on the average, a typical search would retrieve almost sixty percent of the relevant material included in the collection, while only about half of the documents handed to the user in response to a search request would be nonrelevant. If one considers that the MEDLARS col-

lection consists of a half-rfiillion documents, and that only a few hundred will, on the average, prove relevant to a given query, it is seen that the search system consistently and properly rejects many hundreds of thousands of nonrelevant items which the user obviously does not care to see, while retrieving, at the same ,time, a large proportion of the useful items. The operating ranges for the present MEDLARS system are shown in the recall-precision graph of Fig. 2. In practice, the system

operates in the center of the curve, since that is the area where neither recall nor precision are unreasonably low. Other operating areas are, of

course, possible by sliding up and down the curve of Fig. 2. However, most users are not likely to prefer either the high precision — low recall or the low precision — high recall ends, particularly since all points within easy reach of the presently implemented system are quite

VI-10

Precision
ii

o

o MEDLARS Indexing (complete) ideal system

10 . 0 8.
\

0 6. 0 4. 0.2
0 0.2

V

V

°\
\

0.4

0.6

0.8

JL

IJO*"

RecaM

MEDLARS Recall-Precision Range for Complete Test (302 queries, 500,000 documents)
Fig. 2

Percentage of Source of Failure 797 Recall | 3038 Precision Failures Failures

Index Language (lack of specific term or false coordination of terms) Searching (search formulation too exhaustive or too specific) Indexing (document indexing insufficiently exhaustive, or too exhaustive, or omission of important term) Lack of User-System Interaction Miscellaneous

10%

36%

35%

32%

37% 25% 1%

13% 17% 2%

Typical Recall and Precision Failures for Complete MEDLARS Test (302 queries, 500,000 documents) Table 1

VI-11

far away from the ideal operating range in the upper right-hand corner of the curve. An indication of the recall and precision failures identified during the complete MEDLARS test is given in Table 1. It is seen that

over thirty percent of both the recall and the precision failures are due to the fact that the manual query formulation does not adequately reflect the real user need. In addition, the indexing language in use produces

many precision failures, and the document indexing is responsible for many recall failures. Finally, the lack of communication between user and system personnel during the search also causes a large number of errors. A comparison of these test results with those applicable to the SMART runs is made following the exposition of the test design actually used. B). Design of the SMART Test

For a variety of reasons, having to do mostly with input keypunching, it was necessary to restrict the SMART tests to a small subset of the total MEDLARS test environment (300 queries, several thousand recall and precision base items, and over half-a-million documents to be searched). Specifically, eighteen queries were obtained from the National Library of Medicine, together with 273 of their associated RB and PB documents. The 273 documents actually used were chosen as follows: Total documents evaluated by MEDLARS for the 18 queries in SMART subcollection (including 149 RB and 369 PB items) Number unusable for SMART experiment because abstract or summary was not easily available 245 518

VI-12

Total documents used in SMART subcollection

273

For the remaining 18 queries and 273 documents, the English abstracts were keypunched and the SMART runs were carried out in accordance with the standard SMART methods [1,2,3,5]. In order to make a comparison with the MEDLARS system possible, it was necessary, in addition, to choose a cutoff in the number of retrieved documents equivalent to that which MEDLARS would have obtained, had the SMART subcollection been used during the MEDLARS search. A typical cutoff computation is shown as an example in Table 2. Consider a typical request for which MEDLARS would have retrieved a total of 213 documents, including three RB items out of a total of six, thirty PB items, and 180 items retrieved but unassessed for relevance with respect to the given query. If it is assumed

that the SMART subset contains all of the RB items, and twenty out of thirty PB items, then the retrieval cutoff is set at 23 documents for the recall calculations. (Since the SMART system ranks documents in decreasing

correlation order with the search request, it is always possible to retrieve exactly the n highest ranking itemsl. For the precision calculations, an additional adjustment is necessary since recall-base documents which are retrieved by MEDLARS are normally excluded from the MEDLARS precision calculations. For the example of Table 2, three such RB items had to be removed,

the final cutoff being then 20 for precision purposes. This procedure for (determining the number of documents to be retrieved by SMART permits a direct comparison with the MEDLARS searches for the eighteen queries being processed. The following differences in the test

VI-13

A)

Recall Computation Retrieved by MEDLARS 3 30 180 RB (out of 6) PB unassessed

Documents contained in SMART Subset

all 20 20

RB PB PB + 3 RB = 23

SMART Cutoff Assuming 4 RB retrieved (out of 23 items)

Recall 4/6 = 0 . 6 6

B)

Precision Computation Initial Cutoff Intermediate Cutoff (after removal of RB) Number retrieved by SMART in top 20 Final Cutoff 23 23 - 3 = 20

4

RB, 9 PB relevant

20 (remove 4 RB and replace by 4 new items)

SMART retrieval 9 relevant out of 16 1 relevant out of 4 new items

Precision

9+1 20

0.50

Sample SMART-MEDLARS Cutoff Computation Table 2

VI-14

environments must, however, be noted:

a)

The original MEDLARS searches were conducted, using the complete MEDLARS document collection, whereas the SMART searches were made with the subset for which keypunched abstracts were available; the possible effect of this reduction in collection size is discussed in the concluding section of this study;

b)

The recall and precision bases used for the eighteen queries were larger for MEDLARS C518 items) than for SMART (273 items); the average MEDLARS recall-precision results of Table 4(c) show, however, that the MEDLARS performance for the two document subsets is comparable — indeed, MEDLARS obtains somewhat better results with the smaller set of 273 items — so that no further bias is introduced by the reduction in the size of recall and precision bases;

c)

Since the MEDLARS precision calculations are based on the exact set of documents retrieved by MEDLARS in response to each search request, a comparison with the precision obtained by SMART can be made directly only if SMART retrieves exactly the same items as MEDLARS; in that case, the precision values are the same for the two systems; under all other circumstances, SMART retrieves items not also retrieved by MEDLARS, in which case, the corresponding documents are not normally assessed for relevance by MEDLARS, and a direct precision comparison becomes impossible; a precision adjustment must then be made before the respective values are comparable.

The precision adjustment actually made is based on the following argument; The apparent precision obtained by SMART takes into account only

VI-15

those documents which are retrieved by MEDLAR'S.

But, MEDLARS does

not retrieve all relevant items in its searches — in fact, the MEDLARS recall for the 18 test queries is only 0.64. Thus, the apparent SMART

precision is based on the availability of only sixty-four percent of the relevant documents. The assumption is then made that the SMART pre-

cision would remain the same, were the full set of relevant documents to enter into the computation, instead of the sixty-four percent actually

used; that is, it is conjectured that the proportion of relevant items retrieved would be the same for the unavailable relevant items (those not in the MEDLARS precision base) as for the available ones. Since

Adjusted P 100%

"

Apparent P 64%

'

the adjusted precision is obtained by multiplying the apparent precision by the factor 1.56. The complete argument is summarized in Table 3.

To summarize, the search results obtained by MEDLARS and SMART for the 18 queries are compared in the following manner: The cutoff

value used by SMART to distinguish retrieved from nonretrieved items is exactly the one used in the corresponding MEDLARS search for the subset of 273 items; the recall calculations are based on the retrieval of the complete set of known relevant items, and the output values which result are directly comparable; the apparent precision calculations are based on an average MEDLARS recall of only sixty-four percent, and a suitable adjustment is made to account for the lack of relevance assessments in that part of the collection which is not retrieved by MEDLARS.

VI-16

Precision Adjustment 1. SMART can reach MEDLARS precision value only if it retrieves exactly the same items as MEDLARS (since calculations are based on PB) SMART PB base consists only of items retrieved by MEDLARS, and MEDLARS does not retrieve all relevant (MEDLARS recall on SMART subset = 0.64) SMART and MEDLARS are independent systems, and assuming all relevant were available in SMART collection, some of them would be retrieved by SMART Assuming that the percentage of relevant retrieved were to remain the same if all relevant were available to SMART Apparent Precision Percent relevant in SMART collection (64%) Adjustment Adjusted P Apparent P 100 64 Adjusted Precision All relevant (100%)

2.

3.

4.

Explanation for Precision Adjustment

Table 3

VI-17

SMART (standard) Word Form 1 "Micro" Average "Macro" Average 0.644 0.704 Thes. 0.632 0.695 Word Stem 0.655 0.718

J j

SMART (negative delete) 1 Word Word Thes. Stem Form 0.667 0.665 0.644 0.700 0.667 0.718

I

SMART (upweight) j Word Form 0.770 0.802 Thes. 0.690 0.692 Word ] Stem 1 0.770 0.799

a)

SMART Recall Averages (18 queries, 273 documents)

SMART (standard) Word Form Apparent I Apparent Adjusted Adjusted "Micro" "Macro" "Micro" "Macro" 0.395 0.368 0.583 0.571 Thes. 0.410 0.393 0.605 0.611 Word Stem

SMART (negative |T~ SMART (upweight) delete) I Word I Word Word Word Thes. Thes. Stem Form Form Stem 0.355 0.353 0.524 0.549 0.395 0.394 0.583 0.613 0.385 0.342 0.568 0.53l| 0.445 0.431 0.656 0.670 0.440 0.430 0.649 0.669 0.430 0.421 0.634 0.655

0.389 0.367 0.574 0.570 j

b)

SMART Precision Averages (18 queries, 273 documents)

Recall Micro - average Macro - average (273 documents) Micro - average Macro - average (518 documents) 0.678 0.643

Precision | 0.640 0.625

0.671 0.558

0.568 0.573

c)

MEDLARS Recall - Precision Values (18 queries)

Recall-Precision Tables Table 4

Analysis Methods being Compared A. B. MEDLARS Search SMART Word Stem Recall Apparent Precision Adjusted Precision

Probabilities (A over B) t-Test Sign Test

Comparisons favoring A, B, even

0.5676 0.0001 0.2706

1.0000 0.0002 0.6291

A: B: even:

11 13 12

A. B.

MEDLARS Search SMART Thesaurus Recall Apparent Precision Adjusted Precision 0.6883 0.0004 0.8139 1.0000 0.0005 1.0000 A: B: even: 9 15 12

A. B.

MEDLARS Search SMART Word Form Recall Apparent Precision Adjusted Precision 0.5675 0.0001 0.2911 1 1.0000 1 0.0002 1 0.6291 A: B: even: 12 13 11

Significance Computations for SMART-MEDLARS Comparisons (18 queries, 273 documents, standard runs)

Table 5

VI-19

4.

SMART-MEDLARS Comparison The average recall and precision values obtained for the SMART

and MEDLARS systems are shown in Table 4.

The corresponding statistical Tables 4(a) and 4(b)

significance calculations are given in Table 5,

include, respectively, average recall and average precision values for the SMART runs, for each of three different language analysis systems: The

word form dictionary, which makes it possible to match text words differing only in a final
f

s ' (that is, singular and plural forms of the same word);

the word stem dictionary, which includes a single entry for all words exhibiting the same word stem; and the thesaurus, which is used to recognize also words included within a single thesaurus class (such as synonyms and other related items). For each dictionary system, three different SMART query sets are used for experimental purposes, termed "standard run", "negative delete", and "upweight"f respectively. examined in the next section. well as macro-averages. The results for the last two runs are In each case, micro-averages are given as

The former represent the averages obtained by

comparing the total number of relevant retrieved over all 18 queries, to the total relevant or the total retrieved for all the queries; the latter are the actual per query averages and are normally more representative of the performance experienced by the average user [10]. A comparison of the average recall values for the 18 queries (Tables 4(a) and 4(c)) indicates that the micro-averages slightly favor MEDLARS, whereas the macro-averages slightly favor SMART. That is, MEDLARS

is able to retrieve slightly more relevant documents overall, but SMART

VI-20

exhibits the better average recall per query.

An examination of the recall

values obtained for the individual queries listed in Table 6, reveals that MEDLARS may do very well, or very badly, in retrieving relevant documents, whereas SMART is more consistent in obtaining a performance which is generally neither perfect, nor very poor. Thus, perfect recall is obtained In exchange, MEDLARS

ten times by MEDLARS, but only seven times by SMART.

retrieves not a single relevant document in four instances, where this never happens for the SMART searches. These figures point to a fundamental difference between manual indexing systems, and the automatic text processing schemes used in SMART: Often, the human intermediary charged with the formulation of the search statement in the manual system is exceptionally clever in determining the user's information needs; at other times, however, these needs are misunderstood, thus accounting for the searches with zero recall. In addition,

the manual indexing system is, of course, highly dependent on the richness and completeness of the indexing language, and on the thoroughness and accuracy with which the document indexing is performed. In the automatic text analysis, on the other hand, the complete text of a document abstract is normally used for analysis purposes, and it is very rare indeed that the resulting content identifiers do not reflect the actual document content at least to some extent. In addition,

the automatic environment makes it possible to use complex weighting and matching procedures designed to increase the effect of certain important content identifiers at the expense of others that are less crucial. the same time, the basic dependence on the initial vocabulary is also At

VI-21

responsible for the fact that some relevant items are difficult to retrieve, thus accounting for the less than perfect performance of the SMART searches. The statistical significance output of Table 5 shows clearly that the recall differences between SMART and MEDLARS are not statistically significant; indeed, the sign test probabilities are equal to 1 for each dictionary. Thus, the average recall performance is just about identical

for the two systems. The precision figures for the standard SMART runs and the MEDLARS searches are contained in Tables 4(b) and 4(c) respectively. As expected,

the apparent SMART precision is much smaller than the corresponding MEDLARS precision. However, when the adjustment factor is included, it is seen

that the adjusted precision is only slightly lower for SMART than for MEDLARS, the significant. Overall, the average performance data of Table 4 lead to the conclusion that the MEDLARS and SMART performance is comparable for the 18 queries, with SMART showing a slightly better recall while MEDLARS exhibits a somewhat higher precision. One factor, not taken into account in the average performance figures of Table 4, is the ability of the SMART system to rank the documents in decreasing correlation order with the search requests. Thus, in the differences in performance being again not statistically

comparison with MEDLARS, no distinction is made between different rankings of relevant documents that are retrieved.* Such a ranking is, however,

Actually, a limited system of nested ranking on three levels is available in MEDLARS by constructing three increasingly refined formulations of each search query, thereby producing three nested sets of output documents.

VI-22

important to a user interested in retrieving the relevant items ahead of the nonrelevant ones. To test the ranking ability of the automatic SMART process, a separate test was therefore made by comparing the "rank recall" measure [10] computed from the SMART ranks, with the rank recall obtained from a handranked output list produced manually within the National Library of Medicine for test purposes. The hand-ranked output lists were available for four-

teen of the eighteen queries, shown in the evaluation output of Table 7. It is seen here again that the performance of the two ranking systems — manual ranking with MEDLARS and automatic SMART ranking — is about equally effective, the average rank recall being slightly better for SMART than for MEDLARS. A number of conclusions to be drawn from the foregoing test results are examined following the comparison of the various SMART runs.

5.

Comparison of SMART Analysis Methods Several different language analysis procedures were used for the

SMART runs conducted with the MEDLARS subcollection.

Specifically, runs

were made using document titles only or full document abstracts, and three different dictionaries — known respectively as the word form, word stem, and thesaurus dictionaries — were used for language normalization. The first two dictionaries were generated by machine using for this purpose the standard SMART procedures; the thesaurus was generated by hand [11]. The differences in the dictionary makeup account, in general for the differences in performance observed in the recall and precision measures

VI-23

Recall Range 1.0 0.99-0.50 0.49-0.01 0.0

Number of Queries in Range I MEDLARS SMART (stem) 10 1 3 4 7 7 4 0

Query Distribution in Various Recall Ranges Table 6

1 Query ; No. of PB Relevant 1 Number 02 04 05 06 07 09 10 13 14 16 18 32 40 187 5 13 10 8 3 11 11 10 2 4 2 15 3 9

Rank Recall MEDLARS SMART 0.88 0.91 0.92 0.40 0.43 0.97 0.90 0.97 1.0 0.71 0.60 0.75 1.0 0.70 0.80 1 0.94 1 0.83 1 0.98 0.76 1.0 0.93 0.83 0.63 1.0 0.71 1.0 0.75 1.0 0.69
i

No. of Queries favoring 1 MEDLARS SMART Neither ] X X X X X X X X X X X X X X 5 5 4

Average Rank Recall

0.86

Rank Recall Comparison for "Handranked" MEDLARS with SMART Word Stem Process (14 queries using PB documents only) Table 7

VI-24

of Table 4.

Both the recall and precision values are nearly the same

for word form and word stem dictionaries in the standard SMART runs. The thesaurus dictionary, on the other hand, which would normally be expected to produce better results than either of the suffix dictionaries produces only slightly better precision but slightly worse recall. The thesaurus groupings were actually constructed by a staff member without special knowledge of the medical terminology, and the corresponding performance is thus not typical of the thesaurus results obtained by SMART with document collections in different subject fields [5]. The results obtained by using document titles instead of full abstracts for analysis purposes are, however, fully in accord with comparable data previously obtained for different subject areas. The graph of Fig. 3(a) shows, in particular, that titles are much less effective than abstracts, particularly at the high recall end of the curve. Furthermore, the significance output of Fig. 3(c) indicates that the performance differences are, in fact, statistically significant, at least for the global measures. Two additional minor modifications were made in the query set for test purposes. The first consisted in removing from the query statements all negative phrases included to denote what the user did not wish to retrieve. Such negative phrases are not presently recognized by the stan-

dard SMART analysis methods, with the result that negative statements are actually interpreted as positive subject descriptions. The second query

modification consisted in repeating in the query statements certain technical words occurring in the collection with low frequency. This modification

VI-25

produces an increased weight for the corresponding document identifiers, and may thereby generate a query statement which matches the user's interests more closely than the original. The several types of query formulations resulting from the modification procedure are shown for three queries in Table 8. Both of the query alterations were performed manually, although machine programs might have been written to accomplish the same tasks. The new queries represent formulations that could realistically occur in an environment of informed users who would be instructed not to use negative subject descriptions, and who would emphasize the important technical terms in their query formulation. The evaluation results for "negative deletion" and "upweighting" are included in Table 4 for the recall and precision averages corresponding to the MEDLARS searches, and in Fig. 4 in the form of recall-precision graphs. It is seen that the upweighting process improves both recall and

precision by five to ten percent over the complete range of the recallprecision curve. The negative phrase deletion does not, however, exhibit the same uniformly beneficial effects, although some improvement in precision is noticeable at the low recall end of the curve. The significance data of Fig. 4(c) show that the changes in search effectiveness between original and altered queries are not sufficiently pronounced to be statistically significant. An examination of the search results for the individual queries shows that the negative phrase deletion does not perform equally well for all queries. In particular, the procedure fails to improve the retrieval

if the deletion process reduces the query to only a very short statement, no longer representative of user needs. It may also fail in cases where a

VI-26

Precision

1
1.0
&

-o Word stem abstracts -A Word stem litres

.8

\

fc8>

*5

\N \.N>.
A

A

v\
8

.2h

V\
\ ;

^ 1.0

RecoM

a)

Recall-Precision Graph

Word Stem Abstracts R 0. 0. 0. 0. 0, 0. 0. 0.8 0.9 1.0 RNK LOG NOR NOR REC= PRE= REC= PRE= 0.9167 0.8132 0.8008 0.7646 0.7311 0.6947 0.5893 0.4904 0.3962 0.2814 0.4092 0.6926 0.9104 0.8199

Word Stem Titles

Evaluation Measures Precision at R = 0.1 R = 0.3 R = 0.5 R = 0.7 R = 0.9 RNK LOG NOR NOR REC PRE REC PRE

Probab ilities t-Test Sign Test (Abstracts over Titles)

0.8 0.9 1.0 RNK LOG NOR NOR REC= PRE= REC= PRE=

0.8973 0.8154 0.7610 0.7178 0.6718 0.5694 0.4683 0.3904 0.2909 0.2319 0.3023 0.6446 0.7508 0.7169

0.5066 0.3076 0.2311 0.0438 0.1305 0.1446 0.4790 0.0005 0.0085 0.0000

1.0000 0.2266 1.0000 0.0213 0.1435 0.0042 0.0127 0.0074 0.0127 0.0000

Combined

b)

Recall-Precision Tables

c)

Significance Output

Recall-Precision Data for Abstract-Title Comparisons (MEDLARS collection, 18 queries, 273 documents) Fig. 3

VI-27 Precision a—o Query with negative phrase deletion A—A Original query (abstract word stem)

4
Precision o—o Query with weighted important words A—A Original query (abstract word stem) 1.0 .8 * » a

1.0 .8 .6 .4 .2

£C£-A J
^a^

.6

^s
\ .2 a) .4

.4

is

.2h
.6 .8 •hr*- Recall

s\
.8 1.0 -*~Recolf

Recall-Precision Graphs for Comparison of Original Queries with Altered Queries

Evaluation Measures

Upweight over Original (Sign Test)

Original over Negative (Sign TestM

Measures Precision at R = 0.1 R = 0.3 R = 0.5 R = 0.7 R = 0.9 RNK REC LOG PRE NOR REC NOR PRE b)

Original

Upweight

Negative

0.9167 0.8008 0.7311 0.5893 0.3962 0.4092 0.6926 0.9104 0.8199

0,9627 0.8500 0.8004 0.6441 0.4776 0.4546 0.7260 0.9200 0.8476

0.9288 0.7939 0.7246 0.5794 0.3898 0.3973 0.6874 0.8294 0.7674

Precision at R = 0.1 R = 0.3 R = 0.5 R = 0.7 R = 0.9 RNK REC LOG PRE NOR REC NOR PRE Combined cl

0.6875 1.0000 0.7539 1.0000 0.0574 0.4545 0.1796 0.4240 0.3018 0.0011

1.0000 1.0000 1.0000 0.7266 0.7539 0.7539 0.5078 0.7539 0.7539 0.1215

Recall-Precision Tables

Significance Output

Recall-Precision Comparisons for Original Queries and Altered Queries by Negative Phrase Deletion and Upweighting (MEDLARS collection, 18 queries, 273 documents, abstract stem process) Fig. 4

VI-28

given thesaurus grouping includes a variety of different concepts, where some of these concepts occur in a negative phrase, while others occur in a positive sense within the same query. In that case, the deletion of the

negative phrases produces a decrease in the weight of important terms, which may consequently reduce the search effectiveness. The upweighting process for important technical terms generally produces an improvement in search effectiveness. may be less uniform than expected. However, the improvement

For some queries, it is easy to pick

appropriate terms whose weight should be increased; for example, in query 01, listed in Table 8, the term "lens" may be expected to be much more essential for the subject description than, for example, the term "vertebrate" Other query statements may, however, occur for which the important terms are much more difficult to locate; in such cases, the search improvements due to upweighting may remain small, or may be nonexistent. The two query modification procedures incorporated into the SMART system are only two possible methods which may improve the result of the automatic searches. Similar methods can, of course, also be used for the

semi-manual MEDLARS searches. The prospects for such potential improvements in retrieval effectiveness are taken up in the concluding section.

6.

Conclusions The MEDLARS test comparisons which are described in this study lead

to the same conclusions previously reached in other test environments with the SMART evaluation system [5]: Fully-automatic text analysis and search

VI-29

Query Number 01

Original Query The crystalline lens in vertebrates, including humans, but not drug therapy or surgery. Electron microscopy of lung or bronchi. Pleura or pleural diseases may be excluded.

Negative Phrase Delete The crystalline lens in vertebrates, including humans.

With Term Upweighting [Original Query], crystalline lens, crystalline lens.

03

Electron microscopy of lung or bronchi.

[Original Query], electron, electron, lung, bronchi* 1

13

Blood or urinary steroids in human breast or prostatic neoplasms. Drug therapy, toxicology, etc., to be excluded.

Blood or urinary steroids in human breast or prostatic neoplasm.

[Original Query], steroids, steroids, breast, prostate.

Samples of Query Modification by Negative Phrase Deletion and Upweighting

Table 8

VI-30

systems do not appear to produce a retrieval performance which is inferior to that obtained by conventional systems using manual document indexing and manual search formulations. While the manual indexing and search

formulations can lead to exceptionally fine results when the indexer and/or the searcher are completely aware of the relationships between the stored collection and the user needs, the search results may also be very poor when these conditions are not met. The automatic process, on the other hand, with its exhaustive input data and complex analysis methods performs rarely very poorly, and may often produce completely satisfactory retrieval action. Two important questions mayube asked concerning the practical implications of the foregoing test results: First, is it reasonable to expect that identical results would hold if the automatic text processing methods were applied to the operational MEDLARS environment comprising half-a-million or more documents; and, second, can anything be done to improve the search effectiveness of presently existing automatic and manual information systems beyond those reflected in the recall-precision graphs of Figs. 1 to 4. The first question cannot be answered with full certainty, since it is obviously not likely that keypunched abstracts should ever become available for the full MEDLARS collection. To what extent the present

results can safely be extrapolated to searches performed with the full MEDLARS collection depends to a large extent on whether the set of properly rejected nonrelevant documents included in the MEDLARS collection falls

VI-31

into subject categories which are clearly far away from the query subjects. Obviously, if the nonrelevant documents not included in the SMART subset but included in the full collection could be assumed to be easier to reject than the nonrelevant actually included in the subset, then the SMART results for the full collection should be the same as those obtained for the subset alone. If, on the other hand, there are many more hard-to-reject nonrelevant items in the full collection, than in the subset, the results obtained by SMART on the subcollection may not be directly transferable to those obtainable on the full collection. An estimate for the amount of degradation to be expected

in such a case may be obtained by adding to the SMART subset new documents which are nearly — but not quite — relevant to the search requests, and repeating the searches with the augmented collection. Based on the previous

test results obtained with the SMART system in other subject areas, it is this writer's guess thab the degradation, if any, will be small. assertion remains, however, to be tested. The problem relating to the fundamental improvements of both the SMART and MEDLARS searches is easier to treat. The originators of the This

internal MEDLARS test have, in fact, some pertinent suggestions to make concerning possible changes to be implemented in the search formulations, indexing language, and user-system interaction:

a)

Concerning an appropriate query formulation "...the prime requirement is a complete statement of what the requester is looking for in the requester's own natural language, narrative form; [the query forumation must not] be deliberately phrased.... in a form that the requester believes will approximate a MEDLARS search strategy" [9, p. 117];

VI-3 2

b) Concerning the indexing language to be used "we recommend a shift in emphasis away from the external advisory committee on terminology and towards the continued analysis of the terminological requirements of MEDLARS users as reflected in the demands placed upon the system [9, p. 193]; c) Concerning user-system interaction during the search "the greatest potential for improvement in MEDLARS exists at the interface between user and system; a significant improvement in the statement of requests can raise both the recall and the precision.,.." [9, p. 193]. That these suggestions are all well taken has been shown by the retrieval comparisons previously made with the SMART system [5]. Indeed, the search formulations suggested as ideal for MEDLARS are exactly the ones already used for all SMART searches. Furthermore, the dictionary construction principles derived for the SMART system also point in the direction of greater responsiveness to collection makeup and user needs, and away from committee control [12]. Finally, user-controlled iterative searches have been implemented successfully with the SMART system for several years [13,14,15]. It is difficult to predict exactly how much improvement in search effectiveness may result from the introduction of these various search and retrieval aids. The test results obtained under experimental conditions with

the SMART system appear to indicate that the potential improvement will not exceed ten to fifteen percent, leading to a recall afnd precision performance of 0.70 or 0.75, instead of the present 0.50 to 0.60. would still be far short of what is desirable. Such a performance

However, it is encouraging

VI-33

to note that the present situations are well enough understood to make it reasonable to suggest avenues for the design of future improved systems, including viable automatic search and analysis procedures in place of some of the uncertain manual ones now in use.

VI-34

References

[1]

G. Salton and M. E. Lesk, The SMART Automatic Document Retrieval System — An Illustration, Communications of the ACM, Vol. 8, No. 6, June 1965. M. E. Lesk, Operating Instructions for the SMART Text Processing and Document Retrieval System, Scientific Report No. ISR-11 to the National Science Foundation, Section II, Department of Computer Science, Cornell University, June 1966. G. Salton, et al., Information Storage and Retrieval, Scientific Reports to the National Science Foundation, Nos. ISR-11, ISR-12, ISR-13, Department of Computer Science, Cornell University, June 1966, June 1967, and January 1968. C. W. Cleverdon and E. M. Keen, Factors Determining the Performance of Indexing Systems, Vol. 2 — Test Results, Aslib-Cranfield Research Project, Cranfield, 1966. G. Salton and M. E. Lesk, Computer Evaluation of Indexing and Text Processing, Journal of the ACM, Vol. 15, No. 1, January 1968. A. M. Rees, Evaluation of Information Systems and Services, in Annual Review of Information Science and Technology, Vol. 2, C. Cuadra, editor, Interscience Publishers, New York, 1967. A. M. Rees and D. G. Schultz, A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching, Final Report to the National Science Foundation, Center for Documentation and Communication Research, Case Western Reserve University, October 1967. F. W. Lancaster, Evaluating the Performance of a Large Operating Information Retrieval System, Proceedings of the Second Electronic Information Handling Conference, Thompson Book Company, Washington, 1967. F. W. Lancaster, Evaluation of the Operating Efficiency of MEDLARS, Final Report, National Library of Medicine, January 1968.

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

VI-35

References Ccontd)

[10]

G. Salton, The Evaluation of Computer-based Information Retrieval Systems, Proceedings of the FID 1965 Congress, Spartan Books, Washington 1966, E. M. Keen, Suffix Dictionaries and Thesaurus, Phrase, and Hierarchy Dictionaries, Information Storage and Retrieval, Report No. ISR-13 to the National Science Foundation, Sections VI and VII, Department of Computer Science, Cornell University, January 1968. G. Salton, Information Dissemination and Automatic Information Systems, Proceedings of the IEEE, Vol. 54, No. 12, December 1966. J. J. Rocchio, Jr., Document Retrieval Systems — Optimization and Evaluation, Harvard University Doctoral Thesis, Scientific Report No. ISR-10 to the National Science Foundation, Harvard Computation Laboratory, March 1966. J* J. Rocchio and G. Salton, Information Search Optimi zation and Iterative Retrieval Techniques, Proceedings of the AFIPS Fall Joint Computer Conference, Vol. 27, Spartan Books, November 1965. G. Salton, Search and Retrieval Experiments in RealTime Information Retrieval, Proceedings of the IFIP Congress '68, Edinburgh, August 1968, (also Technical Report 68-8, Department of Computer Science, Cornell University, Ithaca, New York, February 1968).

[11]

[12]

[13]

[14]

[15]

VI-36

Appendix A Recall-Precision Comparisons for Individual Queries

VI-37

^rro LO vo r^- oo LD LO T) g | ^ r ^ <D v D O ^ r o L n o o i n ^ o v o r n o r ^ o r ^ L o o l!5 en
O r H O O O t H r H O O H O O i H O r H O O r H

--1 <* U
0)
i

en
^

i
'

LD

kD ro LO <D oo oo in LO L O O L O o o L o o o L O o o o o o o o o r - o r ^ L o o
O r H O O O r H r H O O r H O O i H O r H O O f H

MART

to

TJ

g

^r ;
O

3 fa ° ^

^r oo LO LO <> ^ r^ro LO LO ^ D O ^ n c N O O i n ^ o ^ n o h O h i / i o
r H O O O r H r H O O r H O O i H O r H O O f H

TJ
>H

g
CD [ ^ O O ^ r H C N L O C N V O L O r H C N r H r ^ O O O O O O r H i H

TJ g U C D

TJ O -P
C £ D C O

12 U)

s^
m 92 ^3

LO 00 LO rH VD M

o o
oo
V

• •
en C

>
• CD H CN LO 0)

^ + >

2
E-j

o

£ En
TJ g

0)

^ o o i O r H C N L O C N ^ O ^ r H r H r H r ^ o o o o o O i H r H

£j «

kO VD

<J

En
Tl

o ol
g

&\ •

o
-H

M

U)

tf

« £ § w

t^m^iHrHLOCNivDLOrHCNrHr^rorornrHrH fa

3: fa

u u 0 O

^ ^ "ST o
VD

O

o

• o•

M

u
n3
O CD I

Q U 1W C D

< .H C U 3 rd S

SH

on
O O O oo O O O O O

00
oo O

<D LO
vDCNO

2 3 Q

00 00

ft

r*

vD VO

^r

r H r H r H o ' O r H r H O r H f H r H O r H O O O r H O

« ^3

1

s

o

• o•

3
03 1

CD CD

3-3 Q M
W -P S <y
i ^

s %

a5 a3
i H m a > r H O L 0 C N O r H r H 0 0 r H r ^ O C N i H C N O rH rH CD CD

> >
03 03 O U O -H U 03

u
-H 03

i

-P rH rd HJ > -P C D O rH
£r* 0)

£

H n o ^ n ^ i n ( N r i H r i f
rH rH rH

y

) n > ^ n ^ ( N H

tf
M-l

m
0

r ^ O O O O I ^ r H r H C N v Q o o ^ o O ' ^ ' ^ o O r H C O ^ O
CM i H r H r H C N r H C N r H r H rH CN H H

-P 2

°
> i

u C D
01

3

iH<>jfn^LOkOr-oo(T>ooo^rvx»oocNor^oo H H H H H n ^ O O O

rH 00

VI-38

| oooo'sr^rt^ooo
TJ g i 5-1 0 i 0 -P O O O O ^ C ^ C N O r ^ LO PO ^ ( N r > ^ J (M

or^or^rHoocT>iooo
L O l O l O ^ O I ^ O O C N t ^ O C M O O C M v D L D O O ^ r o O O O

o

£ i 12 oo

o o o o o o o o o o o o o o o o o o
TJ g
Q) -P CO

I

•H U <D PM

^

If) C D

co o ID oi ^D o n [ oooLOrHooor1 LO in m ^ kD ^

CN

n o n h O ^ h ^ L n i / ) Goooovfloj^oji^oo
O L O O V £ > ^ < £ > L O O O O O

I'D (D

U 0 £

^ o r^ r^ UO

LO

« • o o , ^ :«-« ! o |rH
KQ VD O

x:
H

MAR

H

i

o o o o o o o o o o o o o o o o o o

+ > 0)
[•n
ITJ

2

00

T3 g 0 O £ P-.

r ^ o o o o ^ t ^ - o o o ^ T o o o o c T i C N i o t ^ ^O 0 0 0 0 CM [ ^ ^ t 1 CM O O O O O O O O O

c D O h H n ^ m c o L o r M L n ^ D r ^ o o r M r - o C M ^ C M ^ L O O O ^ O O O O O O O O O O O O O

o

< C

W a) ^
EH

• O•
rH

CO T J 5H

g U

00 00 ID

15 PM
T5 U 0 £ g (D -P CO

ocM^LnooconoooLnrooj^rH(Tin\ro H
TJ g
JH <D O -P 12 CO
k—

-*->
£
fd

0^ 00 00

\D\ 00

M

> <D
.H

U) Q)

< u
En

,C
EH

O o O L O r ^ r - o O o o O r H r ^ r H C M O O C M r H O O L O O rH rH

-P £ <D

. . o o
—.

g o u
G
-H -H O

^ fd
TJ O g O H C M 0 0 L O C 0 0 0 0 0 O 0 0 ^ 0 0 0 0 M ^ r H < T > 0 0 ^ O rH

~~\
00 I

gj| 10

tf

Q* w 1 ft ^ i \< x: £i c*
CO T j

f-l

o

GM

^ 1 00

riso
fd Qj

0

0

M LO * o• j o

G

o w
CN 1

^

• .1 o o

!5 fc

ft

< u n

*c
0) rH ,q fd

tq
g |

^

00 « r< ^

0 -H (/) -H

(

Q
S

o

!

(NooooLncT>ooor^r^vx)r-r^^rr^^LOCN oooor-v^oor^^r-ioo^^r-i^DrHr^a^ oooor^r-c^^cNrHO^r^^^r^^r^oo^ o o o d o o o ' o d o o o o o o o o o

i ^H
CO

cu
-p £ fd

^

§ 1
LHD L o r - r H o ro o o o o i — ir r - i H o o c N i n c MrL o n o ^ o o H H r H r H

CO P^

Q H W <D

S

*
TJ C D > o) -H

I rH fd -P

EH

o

-P tf

n

r - ^ D C ^ r ^ r H O r H ^ O C N J ^ C N O O r - o O i H O O O O O rH r H r H CM H r H r H r H CM H r H

0>

> i

1

<D

^

fHcsioo^rLO^Dr^oocTiooo^r^ooocNjor^oo
H H H H H n ^ C

3
CH

O O r H 00

icro" average aero" average

< CD > J

S S

asic
PQ

i

^ ^ Q

tf

^ CN VO ^ 0 • •1 a o

°

LO

T-ME «
<i

£

u o

u\ o

LO 0 0 G> ^ 0 00 00

• o o

*J

B ^ Q

<J

H

2

CO

VI-39

T$
JH

g
<D

0 -P 12 tn
fd

m ^o n o in u) r- co LO LO r ^ O L o r o L n o o L o ^ o ^ r o o r ^ o r ^ L n o
O r H O O O r H r H O O H O O r H O r H O O H

Rec

(D
EH

w

^r ^ m o in KD en en LO LO vo o LO m LO o o LO o n o r o m o r - o i ^ - L O O
O r H O O O r H r H O O r H O O r H O r H O O H

k
1 CO

^

TJ
O

£
0

C M

I D n in LO vo ooOLomcNOOLO^

r-ro LO LO ^ c o o r ^ o r ^ L O O
TJ
JH

£ fa

O r H O O O r H r H O O O O O r H O r H O O r H g 0) CO

r*»
^0

CO (Ti CM

£ U <D 0 -P COrOLOrHCNLOCNVDLOrHCSJrHr^rOrOrOrHrH

0 -P

o• o

M1 •

h d £ cn

< u

> ^

- 0) H

in

H tf
l ^ P O i r j i H C N L O C N ^ ^ f r H r H r H r - r o r o r O r H r H

< D JC
H

m

—— ^ o

^ MJ
tf

^ M

a)

£J
C O

^r o <£> M •1 o• o
r^
LO KD \o\ KD VD

o
-H -P CD rH (D Q
0) 0) fd
JH

——
TJ
g

O O Icn IS fa

^

( T t r O L O f H r H L O C M ^ L O O C N r H r - O O P O m r H r H

£ fa

u O u 0

o• o

•1

Hi

a) >
« < J Q W rH rH C U U 0)

en o o o m

o o

en o o o m o

r- LO I D CN o

< c •^ Q

3

CO en\

r>. ^r\
v.0 ^0

-H 4-> fd

fd
EH

r H r H r H O O r H r H O r H r H r H O H O O O r H O

s tf
T3

L-JL

o• o
CD fd

•I

tr>
CD

,C
0) rd rH rd -P -H

a5

<C C D ^ -H Q ^

r H O O C T t r H O L O C N j O r H r H r O r H r ^ O C N r H C N O rH rH

u u > >
fd fd

o

S

05

+J •H 05 fd > -P C D 0 rH EH < U

•H H r o C T i r o ^ r m f M H H H r O f O h ^ n ^ f N H rH rH rH 1

o u o o
fd

G

tf
MH MH -P U

o
3

r^oooor^rHrH(NvDrO'^1ro^^rrorHcx),^,0
CN rH rH rH CM rH (N r l H r-\ C M

rH rH

>i

I

^ <D 3 01

1

f H r ^ f ^ ^ L O ^ r ^ C D C ^ o r o ^ v D O O c N O r ^ r o H H H H H r o ^ O O O •H en

ord tern 1

\
O 00 LO (Ti CM r^ L O C O H l D h n C M h O

I

h o

oo i n OJ h o o o o

^r ( N o o o o

c N o j ^ ^ i n r o ^ r o n o o o o o o o o o
r • • LO CT> LO

h§ to.

ant Word Stem

T Rel

, OJ

are

si TJ O
rH

«

CO

MEDL Rele

UOTS
•H O
OJ 5^

^

co

U)

I

VD
O

^rrMvD
^ rH 00

on
r»

ooo^r-r^crir^^mLO
0 0 ( N V O v £ > C M M ) C N r > C 0

\ x
, H

r^Ln^r^r^D^oNi 1 o o o o o o o o

O ^ I H V O ^ ^ O L O O O O O o o o o o o o o o o
T5 0)

TJ
£

5H CD

g

rLO

°M

0 -P
CO

•[ • [ I o O

4J
U)

T5 £

rH 0

fa

P 0

6

^ o o ^ r ^ r ^ oo ooo^rcTicsj r[^oo^rojr-'srog o o o o o o o o

LO CM o o o

r^iHoocr>LOOo uo^)r^oocNjr^o CM < > LO OO ^r OO OO £ o o o o o o o

r 4
00 | Cfi OJ ! 00

[•f-\ T)

2

P ! ^ F-"
P
co T5
5H

i

x

oo M LD k o •[ O• o ^r !c^ CM k r !
LO LOI

o
-H

+J
OJ

rH 0) Q

|

o j f N L n m o o c o n o r o ^ i n c N ^ i H t T i r o ^ o

g
rH

0 £ cMoo^r^r^ooooOrHvDCMCMoocMrHooLOO -H rH -P £ T$
M £

O fa g
OJ CO

o• o

l°l

*I

e n rd U X
OJ

OJ

>

U) OJ

x:
rH

LO (Ti 00 00 ; 0 0 l

0 -P

o•

O

•J

>
•H

<
rd

g
5H

o j o j ^ L n o o c o n o r o o n o j ^ H O ^ n ^ o
rH

ft

"
en
CD X rH 1

£

0 fa

<: hi rt

u*

LO CT> 00

^r

I

-P rd tn
0.)

CT»

oo[

o• o

•I

x
-p -H

h§ r°
CO 0

—
TJ g U\ 0 0 3: fa
5H LO LO 00 1

tf
<C

-H
CO

C X ) 0 0 r - ^ D O O r - v D r H C D v D ^ ) r H v D r H r ^ C r >

c\iooooLO(T>ooor^[^<£>tv-r^^rr^<stlLncM

P
W

o
OJ

oooor^r^(i>^ogH(^r-^)vDt^vDr^ro^) O O O O O O O O O O O O O O O O O O

m oo 00} • o. [ o

:< c: o
-H W •H U O J

s
CO r< -P G >

u

!
,

1 Q
iH

3

o ^r KQ\ o

LO CM KOI
m {

pu

m i n h n o o o n H r i H o o c N m c N i n n ^ o
tH rH rH rH rH

L^ (

.

o —J

C D

tr> rp rd rd
>H

e n

0)
OJ

M 0)

rH n3 -P O FH

> OJ -H
5H

nj

> rd >
"o
rH U

r^<vDcr>r^rHOrHvDrM^rcNoor-oorHooooo
rH H r l fM H r H r H r H CM

r H i H

-P
OJ

"b
M -H

o

a 2
OJ

rd

> » ^

01

3

r H O N i o o ^ m ^ r ^ c o c ^ o o o ^ v D o o c M O r - o o H H H H H r O ^ O O O rH 00

VI-41

p1I nu
M !o
j *H

g
CD

vo oo o o LO(N m m in o o m o o m o o o m o o o o o o o r ^ o r ^ o o
| H r H O O O O r H O O r H r H O r H O r H O r H r H

eca

H

-P
C O

£

tf
H

, cn
0)

• j

SMA

«

, -£
! EH

^ o ^ n i n o o m c o o ^ )

^r

^

oo

o

LOCN

r>

o t ^ o r - o

m

m

TJ U

g C D

O r H d o d i H r H O O r H O O r H O H O p H O

O -P -P £ W

& tT
I'd

C

—

o \o\\ r- \a\\ r> r \ • « o p
"

u u

g

H O O L O P O C N O O i n ( T » O O r

<> oo in x

in

oo r

) O

m r

> - O

in r

> 0 0

1 -H o
£ D

c n
C D EH
i

(Tl VD O

o

CM (Ti

"~"

0 O
r H r H O O O f H r H O O r H r H O r H O r H O r H r H

ft

^

° .K •
P
CN

Ej

TJ g MS *H d) (i) 0 -P
CD •H

S M M W O O £ P n
r H o O L n r H C N ^ r c N v 0 a ^ r H o O r H r > o o o o o o c N r H rH

^ O < ^ e r^ l 00
o |o

tf

o

H
•1

.

>

£

C O TJ U

C D C/)
M

g

-P (1)

c n
C D EH [ ^ O O ^ r H C N i n C N i £ > < X > r H C N O r ^ O O O O O O C N O

O -P
C £ D

rt

> I -H
• ^

^ ^ Ico

rd

cn

TJ g
0 0
r H o o m r H r H L n C N v D O r H o O r H r ^ o o o o o o C N r H rH rH

S3

o

cn

EH

x;

a

—J ^r o o ^ r> v£> •] O• o r- m <> <£> x
^0 VD

^ 00 & *-l\ ^o ]>\ • *1 o o
—

c n
C D -H JH C D ^5 Oi

Ej tf < TJ g
S W & rH <C «H vl 03 Q O W 0) ^ O O

TJ
aj P tp •H (D LO 1

£ fc
O O O O O

o* o
in

.I

x; ^ ft

<
Q)
rH

ro

O

O

O

O

O

O

oo
O

O

<D ȣ> CN

m

O 00

X) rd
H

P U M-J
rH rH

r H r H r H O O r H r H O r H r H r H O r H O O O r H O

2

tf
TJ ' rH CTJ

Ti
U £

g
C D W

O -P

m rHl ^> r-• I . o o

o

S5 3-2 Q W

S

ti

M -P < D

r H O O C ^ r H O m C N O r H r H O O r H r ^ O C N r H C N O rH rH

1 "H D* c n
I -H O C D

- — *-*\o 1 CN CT>
1

KD VD O

u EH x:
H U\

•' O j

. J

o a) tf

rd

En
1 ^H

-P H

I

1

-P C D O rH H C D

id ftJ >

£

i
(

w o o £ fa|
r H O O C T l O O ' N r L n C N r H r H r H O O O O r ^ ^ r O O ' ^ C N r H rH rH rH I

S

3 Tl g

*r\

^ o ^ rvO . o• ] o
00 | 00 *T\ KD\ VD • ] O

tf
M-l 4-1

s
Q

3

*!
r ^ O O O O r ^ r H r H C N ^ O O - ^ O O ^ ^ O O r H O O ^ O CN H r l H (N H CN r H r H rH (N rH rH

M • o

0 -P 2 U

o o u u
o o
r H C N 0 0 ^ t
,

> i

u
3

C D OJ

L r ) ^ D r ^ 0 0 ( T » O 0 0 ' ^ v D 0 0 C N O [ ^ 0 0 H H H H H n ^ O O O rH 00

VI-42

^
, JH

gI
CD

I 0 -P c 0
•H 01
•H

CN 0 0 ^ 0 0 0 O O 00 0 0 0 0 LD LD O LD r ^ OOOOLDOOOOOOCN

0 O r ^ 0 O r ^ i H 0 O C T > L D L D 0 0 i D O O < X > r - 0 O C N r ^ 0 0 L D O O O O ^ O L D O O ^ O O O O

o T5
'.;

g
CD

00

rH CN

iJ5 co

o o o o o o o o o o o o o o o o o o

Lc ^ co
tn
-H

-P

0 -P

^ kr • • o jo
o
1 ^

U

CD

^ O,
r^

C O C D £\

I ^ D O ^ D ^ r O O C N
i | O rO LD <T> O O LD I D CN 0 0 ^ T 00 r-1

O O

H h

H in C O

C D
£ Qa C O C D H

1°
|00

O O L D O O r H v D r ^ r ^ O O L D L D C N O O L D ^ l D O O L D

D o

^

1

^

LH

i

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o. o•
1

^H

&! §
CO

IS ^
TJ CNJOO^CNOOOO g , G 00 < T H O O rO N i oooo^^oo^rcM 0 O 15 fa \ o o o o o o o h O ^ n h H n H L n c N ^DCNOOkDr-OOOOI^^D ^D^TOO^OLDOOOOOO'SI1 o o o o o o o o o o o
CO

fe! tf
n3 g
}q ^

O 0 £ fa

^r ^r

m rH 00

o• o

^• I

CO CD -H

U

a)
T5
*H

g
CD

LD 00 00

CN|

Oi TJ
aj •P D^ -H Q J

^ 00
•1

Tl
JH

g
CD

H->
£

0 15

-P C O

LDr^LD^oor^ooor^LD'srcN^rrHcTiooiDO
rH

C D O > ^ -H fd

-P CO

• o
00

o

rd
(D H

tn
CD CO CD X

co
CD

— —1 — LD CTi en ^
00

>,
o j n m L O C O c o c N O ^ h r o H ^ C N f N i r o h O
rH rH

£

X

E-t

o

r1

^

• o .]
ool
LD 00

- — 1
LD

— 1

p o
M-l O -H W -H C J CD U P-.

: < a.

kii pd gj T5
CO

S T5 g C/) H H g U 0
1X4,

LJO

00

o o

U 0

iDCN^rr^ooooooooo^D^rrM^rHoooovDO
rH

^

fa

l

•! o o• 1
r^l
vol
o-) • J

£

£ rsiooooLDcnoooi^r^^Dr^r^^rr^^riDrNj < to C Q
W O O O O r - ^ O O r ^ ^ D H O O v D v D r H ^ D H r ^ C B

TJ g . U C D H O P fd 3: co
-H CT> l-H M O

OV 00 00

c

o o

•1

| C O C D £1

o
C D

oooor^r^cTi^cNHcTir^^Dvor-vDr-oo'X) (DOOOOOOOOOOOOOOOOCD

s 04 ^
C O

-^ £
L D L D r ^ O O O O O O O r H H H O O C M O O C N L D O O C T i O H rH H rH rH H I

^ tf IS ^

&
g u\ 0 fa

•-* <tf o•
cr>
Ol

°

rn OH

ro o

• I

_ LD 00 1 _— '
yD\

tf rd <C > J (D Q H W ^ g tf
TJ CD > C D -rH

T5 en u O £

ro

o• ' ;

•J

CO

rH fd -P EH

J Q h ^ O l h H O H i f l O J ^ C N n h f O r H C O r o O rH r H r H CN H r H H H CN H rH

<:

tf

o

-P

u \

CD

^
>i

1^

<H CN <> <D\ x . • o o

°

ID

o o u u
rHCNjoo^riDv^r^coc^ooo^^DoorNior^oo
-H rd H H H r H r H O O ^ r O O O rH 00

M CD 3 O*

Appendix B Text of MEDLARS Queries*

1

The crystalline lens in vertebrates, including humans, but not drug therapy or surgery. lens. Crystalline lens, crystalline

2

The relationship of blood and cerebrospinal fluid oxygen concentrations or partial pressures. A method of interest is polarography. Blood oxygen, cerebrospinal oxygen. Pleura or plerual

3

Electron microscopy of lung or bronchi. diseases may be excluded.

Electron, electron, lung, bronchi. Lung, bronchial.

4 5

Tissue culture of lung or bronchial neoplasms.

The crossing of fatty acids through the placental barrier. Normal fatty acid levels in placenta and fetus. Fatty acids, placenta.

6

Ventricular septal defect occurring in association with aortic regurgitation; aortic stenosis probably not involved. defect, aortic regurgitation. Septal

7

Radioisotopes in heart scanning.

Mainly used in diagnosis of Technetium,

pericardial effusions. Also used to study tumors, heart enlargement, aneurysms and pericardial thickening. isotopes, radioisotopes. 8 The effects of drugs on the bone marrow of man and animals, specifically the effect of pesticides. Also, the significance of bone marrow changes. Pesticides, bone marrow. 9 The use of induced hypothermia in heart surgery, neurosurgery, head injuries and infectious diseases. Hypothermia, hypothermia. 0 . Neoplasm immunology, excluding plant tumors, granulomas, oncogenic viruses. Neoplasm immunology. RIHSA, radioactive hippurate, cholegraffin are used. Radio-

VI-44

Appendix B (contd)

13

11

Blood or urinary sterioids in human breast or prostatic neoplasms. Drug therapy, toxicology, etc. to be excluded.

Steroids, steroids, breast, prostate. 14 12 Effect of azathioprine on systemic lupus erythematosus, particularly in regard to renal lesions. ioprine. 16 13 Bacillus subtilis phages and genetics, with particular reference to transduction. 18 14 Subtilis, subtilis. Azath, ioprine, azath,

Renal amyloidosis as a complication of tubercolosis and the effects of steroids on this condition. Only the terms

kidney diseases and nephrotic syndrome were selected by the requester. of interest. 32 15 Prednisone and prednisolone are the only steroids Prednisone, prednisolone, renal amyloidosis.

Homonymous hemianopsia in visual aphasia, particularly measurement and assessment. also of interest. Gerstmann's syndrome and agnosia are

Hemianopsia, hemianopsia.

40

16

Separation anxiety in infancy (i.e. up to two years of age) and in preschool children, particularly separation of a child from its mother. Separation anxiety, separation anxiety. Requirements for methods for analysis;

187

17

Nickel in nutrition:

relation with enzyme systems; toxicity of, in humans and laboratory animals; deficiency signs and symptoms; level in various foodstuffs; level in blood and tissues. nickel. 303 18 The toxicity of organic selenium compounds. toxicity. Selenium, selenium, Nickel, nickel,

Underlined words were added to upweight important concepts and used in upweighting experiment.