IX C O N C L U S I O N IX ° * Overview The main body of the review discussed experimental or operational approaches to the different activities involved in automatic indexing, namely document analysis, document description, searching and index language generation0 It is evident that the approaches to both syntax and semantics are, linguistically speaking, not very sophisticated, particularly in operational systems. The methods being tried in question answering research are more ambitious, but their application to large scale document retrieval systems is still some way off. The most striking feature of research in automatic indexing is how little there is of it0 It is difficult to see it having any impact on the massive operational systems and services which are becoming establishedu It is true that there are not many basic ideas which have not been tested in some form or other, but the experiments performed in the last decade are simply an arbitrary choice from the very much larger set which could, and presumably should, be carried out. It is therefore particularly depressing that the experiments which have been done should be, in the main, so unrelated to one another, and that they should typically involve such small test collections and/or such defective evaluation0 A further difficulty is presented by the fact that so few direct comparisons between manual and automatic indexing have been made. Admittedly those that have agree in showing that automatic indexing is not markedly worse than manual, as appears in Section VIII. But it would be dangerous to build too much on such slight foundations. Very broadly, findings for automatic indexing seem to parallel those for manual experiments: for example on the value of syntax. But automatic indexing in some cases offers opportunities which are not always readily available in manual systemss for instance in connection with weighting. Specific conclusions are as follows: lo Syntax In general the syntactic techniques used either to identify important content bearing words or to provide relational information have not been particularly advanced ones. There is no evidence that they are particularly effective for either purpose. 9,1 2, Semantics The statistical approaches used for extraction do not seem very useful; and those tested for classification generation are not obviously advantageous. But statistically based weighting does seem to benefit performance* Some approaches to the treatment of other system components chiefly developed in automatic indexing research appear to be advantageous: for example, normalising matching coefficients which can be expected to generate a quite discriminating ranking. When the results of all the experiments are taken together with operational practice, the specification for an automated system, simpler than some current retrieval systems, but possibly as effective as they are, or at least not conspicuously inferior, emerges. Some parts are provided manually for reasons of convenience rather than principle. As retrieval systems nowadays have names, this one may be called MODESTA. 1* 2. input analysis : abstracts (+titles). : selection of all words not stigmatised by a manual stop list (this may include some 'fluff1 words as well as syntactic operators). : attachment of within-document frequency weights, 3. 4. description index language : vocabulary control through application of suffix stripping. searching : exploitation of further vocabulary control through use of collection frequency based weights. 50 The system would usually operate with a normalising matching coefficient. Its performance might be improved by relevance feedback, and its operation made more convenient by document clustering. It would not be very expensive. IX o 2 Recommendations The specification of MODESTA just outlined is in some ways rather dank: she does not attract. But the all-singing, all-dancing systems so enthusiastically predicted in the early days of mechanised information retrieval are not readily achieved. It is more useful to think of genuine research problems than pie in the sky. In particular, it is evident that we are very far from understanding why many results are obtained. For example, why do statistical association techniques perform so variably? In my view, the real research need is for experiments which will provide us with explanations for performance results, and not merely results. 9,2 A sensible medium range programme would therefore be to carry out rather bigger,, and better designed, experiments over a proper range of collections and test variables. Hopefully, these should give us some information about such factors as exhaustivity, for example, or term cooccurrence patterns, which would help us to understand not merely why automatic techniques behave in observed ways, but (ultimately) retrieval systems in general. A specific point is that vocabulary control in MODESTA is rather rudimentary, and it is doubtful whether these simple methods would be adequate for really large collections, A very worthwhile line would therefore be to investigate simple automatic control techniques for large vocabularies, given that more recognition of verbal relations would seem to be required. 9.3 B I B L I O G R A P H Y The references are in alphabetical order of sole or lead author (see definition in the Preface), and in date order for each author. Abbreviations s I, So & R, Jo Doc J. ASIS = • = Information Storage Retrieval Journal of Documentation Journal of the ASISo Aitchison, JG and Gilchrist, A« Thesaurus Construction; A Practical Manual, London: Aslib, 1972 1970 Aitchison, T0M0, Hall, A.M., Lavelle, K.H, and Tracy, JoM. Comparative Evaluation of Indexing Languages, Part II: Results, Project Inspec, Institute of Electrical Engineers, London, 1970 1971 Andrews, K. Dissertation for the Diploma in Computer Science, Computer Laboratory, University of Cambridge, 1971 Armitage 1967 Armitage, J.E0 and Lynch, MaF. "Articulation in the Generation of Subject Indexes by Computer", Journal of Chemical Documentation, 7, 1967, 170-178 1968 Armitage, J c E e and Lynch, M„F0 "Some Structural Characteristics of Articulated Subject Indexes", I. S, & Rc, 4, 1968, 101-111 1970 Armitage, J C E 0 , Lynch, M.F., Petrie, J.H. and Belton, M„ "Experimental Use of a Program for Computer-Aided SubjectIndex Production", Io S0 & R., 6, 1970, 79-87 Artandi 1969a Artandi, Sc "Computer Indexing of Medical Articles", J, D o c , 25, 1969, 214-223 1969b Artandi, S* and Wolf, E„H. "The Effectiveness of AutomaticallyGenerated Weights and Links in Mechanical Indexing", American Documentation, 20, 1969, 198-202 1968 Austin, C.J. MEDLARS 1963-1967, U,S.Department of Health Education and Welfare, Public Health Service, Washington DCC», 1968 1968 Avram, H,D. The MARC Pilot Project, Final Report, Information Systems Office, Library of Congress, 1968 1973 Barber, A,S 9 , Barraclough, E*Da and Gray, W.A« "On-line Information Retrieval as a Scientists Tool", I» S. & R., 9, 1973, 429-440 Barker 1972a Barker, FoH,, Veal, D.C. and Wyatt, B„K„ "Comparative Efficiency of Searching Titles, Abstracts and Index Terms in a Free-Text Data Base", Jo D o c , 28, 1972, 22-36 1972b Barker, F«H., Veal, D.C. and Wyatt, B.K. "Towards Automatic Profile Construction", JP D o c , 28, 1972, 44-55 1972 B.l 1972 1973 1972 1967 1970 1973 1972 1973 1969 1970 1973 1969 1970 1973 1965 1965 1966 1970 1971 Barlow, D.He "Information Retrieval", The Computer Bulletin, 16, 1972, 250-256 Bath University Library, Design of Information Systems in the Social Sciences Project, "Clustering of Journal Titles According to Citation Data", Working Paper No, 11, September 1973 Bennett, J,L. "The User Interface in Interactive Systems" in Annual Review of Information Science and Technology (Ed* Cuadra), Vol. 7, Washington DaC\s ASIS, 1972 Borko, H. (Edo) Automated Language Processing, New York: John Wiley, 1967 Borkowski Borkowski, C, et al0 "Structure and Effectiveness of *The Citation Identifier7: An Operational Computer Program for Automatic Identification of Case Citations in Legal Literature", J, ASIS, 21f 1970, 8-15 Borkowski, C. and Martin, J0S. "Structure, Costs and Benefits of LEXtractor, An Operational Computer Program for Automatic Extraction of Case Summaries and Dispositions from Court Decisions", Proceedings of the ASIS, Vol. 10, 1973, 21-22 Borman, L. and Mittman, Bo "Interactive Bibliographic Data Bases in an Academic Environment", Jq ASIS, 23, 1972, 164-171 Bousselety p 0 "Le Systeme Titus", Nachrichten„ Nouvclles, Notizie, 49, 1973, 157-163 Brauen, T.L* "Document Vector Modification in On-line Information Retrieval Systems", Information and Storage Retrieval, Report No• ISR-17, Department of Computer Science, Cornell University, 1969, Cagan, C. "A Highly Associative Document Retrieval System", J. ASIS, 21, 1970, 330-337 Campey, L. "A Survey of Index Generation Programs", I, S, & R*, 9, 1973, 441-448 Carroll Carroll, J.M. and Roelloffs, R„ "Computer Selection of Keywords Using Word-Frequency Analysis", American Documentation, 20, 1969, 227-233 Carroll, J.M„ and DeBruyn, J0G, "On the Importance of Root-Stem Truncation in Word-Frequency Analysis", JQ ASIS, 21, 1970, 368-369 Carroll, JcM. and Tague, J.M. "Use of an Automatic Analyser in Preparation of SDI Profiles", J. ASIS, 24, 1973, 277-281 Chomsky, No Aspects of the Theory of Syntax, Cambridge, Mass.: MIT Press, 1965 Clarke, DoC* and Wall, R 9 E 0 "An Economical Program for Limited Parsing of English", AFIPS Conference Proceedings, Vol.27, Pt.l, 1965, 307-316 Cleverdon Cleverdon, C<>W., Mills, J. and Keen, Me Factors Determining the Performance of Indexing Systems, 2 vols. College of Aeronautics, Cranfield, 1966 Cleverdon, C.W, "An Investigation into a Suitable Mechanised Information Retrieval System at the Defence Operational Analysis Establishment", Cranfield Institute of Technology, 1970 Clough, C.R« and Brarawell, K*M0 "A Single Computer-Based System for Both Current Awareness and Retrospective Search: Operating Experience with Assassin", J«, Doc., 27, 1971, 243-253 B.2 Coates, E0JQ "Some Properties of Relationships in the Structure of Indexing Languages8', Jc Doc^ 29, 1973, 390-404 1970 Cooper, W 0 S, "The Potential Usefulness of Catalog Access Points other than Author, Title and Subject", JQ ASIS, 21, 1970, 112-127 Coyaud 1966 Coyaud, M0 Introduction h 1'Etude des Langages Documentaires, Paris: Klincksieck, 1966 1972 Coyaud, MQ Linguistique et Documentation, Paris: Larousse, 1972 Cuadra 1966Cuadra, C0A0 (Ed0) Annual Review of Information Science and Technology, Vols 1-, 19661971 Cuadra, C0A0 "On-line Systems: Promise and Pitfalls", J« ASIS, 22, 1971, 107-114 1965 Damerau, F0J, "An Experiment in Automatic Indexing", American Documentation, 16, 1965, 283-289 1969 Dattola, R 0 T 0 "Experiments with a Fast Algorithm for Automatic Classification", in Information Storage and Retrieval, Report No0 ISR-16, Department of Computer Science, Cornell University, 1969; reprinted in Salton 1971a Dennis 1965 Dennis, SQF0 "The Construction of a Thesaurus Automatically from a Sample of Text" in Statistical Association Methods for Mechanised Documentation (Ed« Stevens et alo), National Bureau of Standards, Washington D o C , 1965 1967 Dennis, S0Fo "The Design and Testing of a Fully Automated IndexingSearching System for Documents Consisting of Expository Text" in Information Retrieval - A Critical View (Ed, Schechter), Washington D 0 C 0 : Thompson, 1967 1966 Doyle, LoB0 and Blankenship, D0A, "Technical Advances in Automatic Classification", Proceedings of the API, Volo3, 1966, 63-71 Earl 1970 Earl, LoL*, "Experiments in Automatic Indexing and Extracting", Ic S0 & Ro, 6, 1970, 313-334 1972 Earl, L0L0 "The Resolution of Syntactic Ambiguity in Automatic Language Processing", IQ So & RQ, 8, 1972, 277-308 1973 Earl, L 0 L 0 "Use of Word Government in Resolving Syntactic and Semantic Ambiguities", I0 Sa & R„, 9, 1973, 639-664 Edmundson 1961 Edmundson, HoPa and Wyllys, R«,E. "Automatic Abstracting and Indexing - Survey and Recommendations", Communications of the ACM, 4, 1961, 226-234 1969 Edmundson, H0Po "New Methods in Automatic Extracting", Journal of the ACM, 16, 1969, 264-285 Fangmeyer 1969 Fangmeyer, Ho and Lustig, G0 "The Euratom Indexing Project" in Information Processing 68 (Ed, Morrell), Amsterdam: NorthHolland, 1969 1970 Fangmeyer, He and Lustig, Go "Experiments with the CETIS Automatic Indexing System" in Handling of Nuclear Information, International Atomic Energy Agency, Vienna, 1970 1973 B«3 Farradane Farradane, J, "Concept Organisation for Information Retrieval", L So & R„, 3, 1967, 297-311 1973 Farradane, J„, Russell, J.Ma and Yates-Mercer, P.A. "Problems in Information Retrieval: Logical Jumps in the Expression of Information", I0 S„ & Ro, 9, 1973, 65-77 1973 Gardin, J0-C0 "Document Analysis and Linguistic Theory", J. Doc«, 29, 1973, 137-168 1972 Gerson, G.M„ "A Computer-Aided Classification and Retrieval System for United States Patents", I« S« & Rc, 8, 1972, 137-150 1968 Gotlieb, CCo and Kumar, So "Semantic Clustering of Index Terms", Journal of the ACM, 15, 1968, 493-513 1973 Hansen, I«B,3 "CA Condensates as a Retrospective Search Tool. A Commentary", I. S, & R„, 9, 1973, 201-205 Hillman 1968 Hillman, D.Ja "Negotiation of Inquiries in an On-line Retrieval System", I0 SQ & R., 4, 1968, 219-238 1969 Hillman, D.Je and Kasarda, A 0 J 0 "The LEADER Retrieval System", AFIPS Conference Proceedings, VolQ34, 1969, 447-455 1973 Hillman, D0J0 "Customerised User Services via Interactions with LEADERMART", I« So & R,, 9, 1973, 587-596 1971 Hines, T4CC and Harris, J0L* "Columbia University School of Library Service System for Thesaurus Development and Maintenance", L So & Ro, 7, 1971, 39-50 1971 Hisinger, B« "A Multidisciplinary and Computerised SDI Service for Industry and Research - Practical Experience and Costs", Io So & R., 7, 1971, 175-187 1973 Hoyle, W.Co "Automatic Indexing and Generation of Classification Systems by Algorithm", Ic S, & R«, 9, 1973, 233-242 1969 Ide, Eo "Relevance Feedback in an Automatic Document Retrieval System", Information Storage and Retrieval, Report No„ISR-15, Department of Computer Science, Cornell University, 1969 IJCAI = Proceedings of the International Joint Conference on Artificial Intelligence 1969 First, The Mitre Corporation, Bedford, Mass«, 1969 1971 Second, The British Computer Society, London, 1971 1973 Third, Stanford University, Stanford, Calif0, 1973 1966 Ivie, E,L„ Search Procedures Based on Measures of Relatedness Between Documents, Thesis, Massachusetts Institute of Technology, 1966 1973 Jacquesson, AQ and Schieber, W 0 D 0 "Term Association Analysis on a Large File of Bibliographic Data, Using a Highly-Controlled Indexing Vocabulary", I« So & Ro, 9, 1973, 85-94 1971 Jardine, N, and Sibson, R. Mathematical Taxonomy, New York: John Wiley, 1971 Jones 1967a Curtice, R0M0 and Jones, P,E, "Distributional Constraints and the Automatic Selection of an Indexing Vocabulary", Proceedings of the ADIo, Vola4, 1967, 152-156 1967b Jones, P 0 E 0 , and Curtice, RaM0 "A Framework for Comparing Term Association Measures", American Documentation, 18, 1967, 153-161 1967 B«4 1968 1969 1963 1967 1972 1973 1971 1965 1973 1973a 1973b 1972 1965 1966 1968a 1968b 1969 1972a 1972b Jones, P,Eop Curtice, RcM,, Giuliano, V.E. and Sherry,, MflEa "Application of Statistical Association Techniques to the NASA Document Collection", A,D.Little Inc0, Cambridge, Mass,, 1968 Curtice, R0M0 and Jones, P.E* "An Operational Interactive Retrieval System", A,D0Little Inc., Cambridge, Mass., 1969 Katz, J,JC and Fodor, J0Ao "The Structure of a Semantic Theory", Language, 39, 1963, 170-210 Keen Keen, E0M. Several chapters in Information and Storage Retrieval, Report NOo ISR-13, Department of Computer Science, Cornell University, 1967 Keen, EoM. and Digger, J,A* Report of an Information Science Index Languages Test, 2 vols* College of Librarianship Wales, Aberystwyth, 1972 Keen, E*Ma "The Aberystwyth Index Languages Test", Jo Doc0, 29, 1973, 1-35 Kerchner, M,D9 "Dynamic Document Processing in Clustered Collections", Information Storage and Retrieval, Report No. ISR-19, Department of Computer Science, Cornell University, 1971 Kessler, MJ!0 "Comparison of the Results of Bibliographic Coupling and Analytic Subject Indexing", American Documentation, 16, 1965, 223-233 Kimball, J e P 0 The Formal Theory of Grammar, Engleirood Cliffs, N.J.: Prentice-Hall, 1973 Klingbiel Klingbiel, P0H„ "Machine-Aided Indexing of Technical Literature", L S, & R0, 9, 1973, 79-84 Klingbiel, P0H* "A Technique for Machine-Aided Indexing", I, So & Ro, 9, 1973, 477-494 Krevett, B« and Griffith, B8C. "A Comparison of Several Zipf-Type Distributions in their Goodness of Fit to Language Data, J. ASISo 23f 1972, 220-221 Kuno Kuno, S. "The Predictive Analyser and a Path Elimination Technique",, Communications of the ACM, 8, 1965, 453-462 Kuno, S. "Automatic Syntactic Analysis" in Seminar on Computational Linguistics (Ed. Pratt et al0) U.S0Department of Health Education and Welfare, Washington D„C0, 1966 Lancaster Lancaster, F0W. Evaluation of the MEDLARS Demand Search Service, National Library of Medicine, Bethesda, Md., 1968 Lancaster, F0W0 Information Retrieval Systems: Characteristics, Testing and Evaluation, New York: John Wiley, 1968 Lancasterp F.WQ "MEDLARS: A Report on the Evaluation of its Operating Efficiency", American Documentation, 20, 1969, 119-142 Lancaster, FwWtt Vocabulary Control for Information Retrieval, Washington DoC0: Information Resources Press, 1972 Lancaster, F.W., Rapport, R.La and Kiffin Penry, J. "Evaluating the Effectiveness of an On-line, Natural Language Retrieval System", I0 S, & RQ, 8, 1972, 223-245 B.5 1972 1968 1969 1969 1968 1971 1968 1967 1970 1971a 1971b 1970a 1970b 1972 1973 1968 1972 1972 1972 1973 1971 Lefever, M„, Freedman, B, and Schultz, L, "Managing an Uncontrolled Vocabulary Ex Post Facto", Je ASIS, 23, 1972, 339-342 Lesk Lesk, M«Eo "Performance of Automatic Information Systems", L S 0 & R., 4, 1968, 201-218 Lesk, M 0 E 0 "Word-Word Associations in Document Retrieval Systems", American Documentation, 20, 1969, 27-38 Litofsky, B, Utility of Automatic Classification Systems for Information Storage and Retrieval, Thesis, University of Pennsylvania, 1969 Lovins Lovins, J 0 B o "Development of a Stemming Algorithm", Mechanical Translation, 11, 1968, 22-31 Lovins, J 0 B o "Error Evaluation for Stemming Algorithms as Clustering Algorithms", J, ASIS, 22, 1971, 28-40 Lyons, J„ An Introduction to Theoretical Linguistics, Cambridge: Cambridge University Press, 1968 Mathews, W 0 D. "The TIP Retrieval System at MIT" in Information Retrieval - A Critical View (Ed« Schechter), Washington, D.C.: Thompson, 1967 Matthews, F 0 W 0 "Weighted Term Search and Strategy for Searching a Large File for Document Retrieval by Computer", Proceedings of the ASIS, Vol 0 7, 1970, 315-317 Miller Miller, W 0 L o "The Efficiency of MEDLARS Titles for Retrieval", Jo ASIS, 22, 1971, 318-321 Miller, W,L C "A Probabilistic Search Strategy for MEDLARS", J, Doc,, 27, 1971, 254-266 Minker Augustson, J,Go and Minker, Jo "An Analysis of Some Graph Theoretical Cluster Techniques", Journal of the ACM, 17, 1970, 571-588 Augustson, J e G f and Minker, J0 "Deriving Term Relations for a Corpus by Graph Theoretical Techniques", J, ASIS, 21, 1970, 101-111 Minker, J,, Wilson, G*Ae and Zimmerman, B.Ho "An Evaluation of Query Expansion by the Addition of Clustered Terms for a Document Retrieval System", I, S. & R 0 , 8, 1972, 329-348 Minker, J., Peltola, E* and Wilson, G.A* "Document Retrieval Experiments Using Cluster Analysis", J0 ASIS, 24, 1973, 246-257 Minsky, Mo Semantic Information Processing, Cambridge, Mass.: MIT Press, 1968 Montgomery, C G A. "Linguistics and Information Science", J. ASIS, 23, 1972, 195-219 Moody, D«W6 and Kays, 0 o "Development of the US Geological Survey Bibliographic System Using GIPSY", J0 ASIS, 23, 1972, 39-47 Murray, D„M0 "Document Retrieval Based on Clustered Files", Information Storage and ^^rieval, Report No. ISR-20, Department of Computer Science, Cornell University, 1973 Myers, J g H 0 "Computers and the Searching of Law Texts in England and North America: A Review of the State of the Art", Jo D o c , 29, 1973, 212-228 Negus, A*E. and Hall, J 0 L 0 "Towards an Effective On-line Reference Retrieval System", I. S 0 & R», 7, 1971, 249-270 Bo6 1970 1973 1968 1973 1972 1969 1972 1971 1972 1973 1966 1970 1970 1971 1973 1972 1973a 1973b Niblett, G.B.Fo and Price, NoH. "Mechanised Searching of Acts of Parliament", I, SP & R., 6, 1970, 289-297 Olive, G., Terry, J0E. and Datta, So "Studies to Compare Retrieval Using Titles with that Using Index Terms", J0 Doc., 29, 1973, 169-191 Parker, EflBfl "SPIRES (Stanford Public Information Retrieval System)", 1968 Annual Report, Institute for Communication Science, Stanford University, 1968 Petrick, S„RQ "Transformational Analysis" in Natural Language Processing (Ed* Rustin), New York: Algorithmics Press, 1973 Prywes, N.S* and Smith, D.P. "Organisation of Information" in Annual Review of Information Science and Technology (Ed«Cuadra), Vol,7, Washington D,C: ASIS, 1972 Reintjes, JCF. "System Characteristics of Intrex", AFIPS Conference Proceedings, Vol*34, 1969, 457-460; see also adjoining papers* Rettenmayer, JrJW. "File Ordering and Retrieval Cost", I. So & R., 8, 1972, 79-93 van Rijsbergen Jardine, No and van Rijsbergen, C.J, "The Use of Hierarchic Clustering in Information Retrieval", I, S, & R«, 7, 1971, 217-240 van Rijsbergen, C J , Automatic Information Structuring and Retrieval, Thesis, University of Cambridge, 1972 van Rijsbergen, CaJ, and Sparck Jones, K0 "A Test for the Separation of Relevant and Non-relevant Documents in Experimental Retrieval Collections", J. D o c , 29, 1973, 251-257 Rocchio, J,J. "Document Retrieval Systems - Optimisation and Evaluation", Information Storage and Retrieval, Report No. ISR-10, The Computation Laboratory, Harvard University, 1966 Rolling, L.N. "Compilation of Thesauri for Use in Computer Systems", Io S. & R., 6, 1970, 341-350 Rowlands, D„G. "The Unilever Research SDI System", I. S. & R., 6, 1970, 53-71 Rush Rush, J.E., Salvador, R. and Zamora, Ac "Automatic Abstracting and Indexing. II. Production of Indicative Abstracts by Application of Contextual Inference and Syntactic Coherence Criteria", J, ASIS, 22, 1971, 260-274 Mathisp B.A., Rush, J,EC and Young, C.Eo "Improvement of Automatic Abstracts by the Use of Structural Analysis", J, ASIS, 24, 1973, 101-109 Sager Sager, Nc "Syntactic Formatting of Science Information", AFIPS Conference Proceedings, Vol.41, 1972, 791-800 Sager, N. "The String Parser for Scientific Literature", in Natural Language Processing (Ed. Rustin), New York-. Algorithmics Press, 1973 Grishman, R., Sager, N., Raze, Co and Bookchin, Bft "The Linguistic String Parser", AFIPS Conference Proceedings, Vol. 42, 1973, 427-434 B.7 1966- 1968a 1968b 1968c 1968d 1969a 1969b 1969c 1969d 1970a 1970b 1970c 1971a 1971b 1972a 1972b 1972c 1972d 1973a 1973b 1973c Salton Salton, G. (Project Director), Information Storage and Retrieval, Reports Nos* ISR-11, 12, 13, 14, 16, 18, 21, Department of Computer Science, Cornell University, 1966-1972 Salton, Gt Automatic Information Organisation and Retrieval, New York: McGraw-Hill, 1968 Salton, Gc "Search Strategy and the Optimisation of Retrieval Effectiveness" in Mechanised Information Storage, Retrieval and Dissemination (Ed„ Samuelson), Amsterdam: North-Holland, 1968 Salton, G„ and Lesk, MeE« "Computer Evaluation of Indexing and Text Processing", Journal of the ACM, 15, 1968, 8-36 Lesk, M0E« and Salton, Go "Relevance Assessments and Retrieval System Evaluation", I, So & R,, 4, 1968, 343-359 Salton, G. "A Comparison between Manual and Automatic Indexing Methods", American Documentation, 20, 1969, 61-71 Salton, G. "Search and Retrieval Experiments in Real-Time Information Retrieval", Information Processing 68 (Ed. Morrell), Amsterdam: North-Holland, 1969 Lesk, M,E„ and Salton, G* "Interactive Search and Retrieval Methods Using Automatic Information Displays", AFIPS Conference Proceedings, Vol,34, 1969, 435-446 Ide, E. and Salton, G„ "User Controlled File Organisation and Search Strategies", Proceedings of the ASIS, 6, 1969, 183-191 Salton, G 0 "Automatic Text Analysis", Science, 168, 1970, 335-343 Salton, G. "Automatic Processing of Foreign Language Documents", J, ASIS, 21, 1970, 187-194 Salton, G. "Evaluation Problems in Interactive Information Retrieval", L S, & R., 6, 1970, 29-44 Salton, G* (Ed.) The SMART Retrieval System: Experiments in Automatic Document Processing, Englewood Cliffs, N»J 0 : Prentice-Hall, 1971 Salton, G. "Automatic Indexing Using Bibliographic Citations", J. D o c , 27, 1971, 98-110 Salton, G. "Experiments in Automatic Thesaurus Construction for Information Retrieval", in Information Processing 71 (Ed. Freiman), Amsterdam: North-Holland, 1972 Salton, G. "The 'Generality* Effect and the Retrieval Evaluation of Large Collections", J„ ASIS, 23, 1972, 11-22 Salton, G. "A New Comparison between Conventional Indexing (MEDLARS) and Automatic Text Processing (SMART)", Jc ASIS, 23, 1972, 75-84 Salton, G. "Dynamic Document Processing", Communications of the ACM, 15, 1972, 658-668 Salton, Go "Automatic Processing of Current Affairs Queries", I. So & R., 9, 1973, 165-180 Salton, G« "Recent Studies in Automatic Text Analysis and Document Retrieval", Journal of the ACM, 20, 1973, 258-278 Salton, G. and Yang, C.S* "On the Specification of Term Values in Automatic Indexing", Jo D o c , 29, 1973, 351-372 B«8 1968 1971 1972 1971 1968 1965 1970 1963 1969 1964 1967 1970a 1970b 1971a 1971b 1971c 1972 1973a 1973b 1973c Saracevic Comparative Systems Laboratory, An Inquiry into Testing of Information Retrieval Systems, 3 vols. Western Reserve University, 1968 Saracevic, T* "Selected Results from an Inquiry into Testing of Information Retrieval Systems", Jc ASIS, 22, 1971, 126-139 Scheffler, Fc, March, Ja and Bernados, JQ "An Experiment to Study the Use of Boolean NOT Logic to Improve the Precision of Selective Dissemination of Information, J. ASIS, 23, 1972, 58-65 Schiminovich, S« "Automatic Classification and Retrieval of Documents by Means of a Bibliographic Pattern Discovery Algorithm", Ic So & Ro, 6, 1971, 417-435 Schultz, CoKo (Edo)<, HQP,Luhn: Pioneer of Information Science Selected Works, London: Macraillan, 1968 Sharp, J0R0 Some Fundamentals of Information Retrieval, Londons Andre Deutsch, 1965 Simmons, R,F. "Natural Language Question-Answering Systems", Communications of the ACM, 13, 1970, 15-30 Sokal, RGRo and Sneath, P.H.A, Principles of Numerical Taxonomy, London: Freeman, 1963 Sommar, HoG0 and Dennis, DeE. "A New Method of Weighted Term Searching with a Highly Structured Thesaurus", Proceedings of the ASIS, Vol06, 1969, 193-198 Sparck Jones Sparck Jones, Kc Synonymy and Semantic Classification, Thesis, University of Cambridge, 1964 Sparck Jones, Kc and Jackson, D.M, "Current Approaches to Classification and Clump-Finding at the Cambridge Language Research Unit, The Computer Journal, 10, 1967, 29-37 Sparck Jones, Kc "Some Thoughts on Classification for Retrieval", J0 D o c , 26, 1970, 89-101 Sparck Jones, K« and Jackson, D„Me "The Use of AutomaticallyObtained Keyword Classifications for Information Retrieval", Io So & Ro, 5, 1970, 175-201 Sparck Jones, Kc Automatic Keyword Classification for Information Retrieval, London: Butterworths, 1971 Sparck Jones, K« and Barber, E«0, "What Makes an Automatic Keyword Classification Effective?", J, ASIS, 22, 1971, 166-175 Sparck Jones, Kc "The Theory of Clumps" in Encyclopedia of Library and Information Science (Ed„ Kent and Lancour), Vol*5, New York: Marcel Dekker, 1971 Sparck Jones, K„ "A Statistical Interpretation of Term Specifity and its Application in Retrieval", Jc Doc*, 28, 1972, 11-21 Sparck Jones, KP and Kay, Mo Linguistics and Information Science, New York: Academic Press, 1973 Sparck Jones, KQ "Index Term Frequency and Quartile Deletion" (Letter), J, ASIS, 24, 1973, 166-167 Sparck Jones, K0 "Collection Properties Influencing Automatic Term Classification Performance", I, S. & R«„ 9, 1973, 499-513 Bo9 Sparck Jones, K. "Does Indexing Exhaustivity Matter?", J, ASIS, 24, 1973, 313-316 1973e Sparck Jones, K. "Index Term Weighting", I, Sc & R», 9, 1973, 619-633 Stevens 1965a Stevens, M.EC Automatic Indexing; A State of the Art Report, Monograph 91, National Bureau of Standards, Washington D,C, 1965 1965b Stevens, M a E 0 , Heilprin, L. and Giuliano, V,E. (Eds,) Statistical Association Methods for Mechanised Documentation, National Bureau of Standards, Washington, D # C G , 1965 Stone 1967a Stone, D.Co Word Statistics in the Generation of Semantic Tools for Information Retrieval Systems, Thesis, University of Pennsylvania, 1967 1967b Rubinoff, MG and Stone, D0C. "Semantic Tools in Information Retrieval", Proceedings of the API, Vol,4, 1967, 169-174 1968 Stone, DoCo and Rubinoff, M0 "Statistical Generation of a Technical Vocabulary", American Documentation, 19, 1968, 411-412 1972 Svenonius, E, "An Experiment in Index Term Frequency", J. ASIS, 23, 1972, 109-121 Syntol 1964 Cros, RQC„, Gardin, J.Ce and Levy, F0 L6Automatisatj.on des Recherches Documentaires. Un Mod&le Generals K e SYNTOL, Paris: f Gauthier Villars, 1964? 2nd Edition, 1968 1965 Gardin, J,C. Syntol, Graduate School of Library Service, Rutgers, The State University, New Brunswick, N«J<., 1965 1967 Coyaud, M. and Siot-Decauville, N. L*Analyse Automatigue des Documents, The Hague: Mouton, 1967 1970 Bely, N., Borillo, Ac, Virbel, J« and Siot-Decauville, N. Procedures d*Analyse Semantique Appliquees a la Documentation Scientifigue, Paris: Gauthier Villars, 1970 Tell 1970 Tell, B.V«, Larsson, R« and Lindh, R« "Information Retrieval with the Abacus Program" in Handling of Nuclear Information, International Atomic Energy Agency, Vienna, 1970 1971 Tell, B0V. "Retrieval Efficiency from Titles and the Cost of Indexing", I, S» & R,, 7, 1971, 241-243 1972 Thiel, L.HU and Heaps, H.S. "Program Design for Retrospective Searches on Large Data Bases", IQ So & R,, 8, 1972, 1-20 1968 Thome, J,P., Bratley, P. and Dewar, Hc "The Syntactic Analysis of English by Machine" in Machine Intelligence 3 (Ed. Michie), New York: American Elsevier, 1968 1970 Vaswani, P.K.T. and Cameron, JeB. The National Physical Laboratory Experiments in Statistical Word Associations and their Use in Retrieval, National Physical Laboratory, Teddington, 1970 1974(?)Vernimb, C O . "Abstracts on Microfiche for On-line Retrieval", New Developments in Storage, Retrieval and Dissemination of Aerospace Information, AGARD, Neuilly sur Seine,, 1974 (?) 1971 Vickery, B„C. "Structure and Function in Retrieval Lanuages", J, Doc, 27, 1971, 69-82 1973d B.10 1973 1969 1972 1971 1973 1968a 1968b 1971 1972 1972 1971 1970 1972 1973 Walker, D.E. "Automated Language Processing" in Annual Review of Information Science and Technology (Ed, Cuadra), Vol.8, Washington D.C.: ASIS, 1973 Weiss, S a F 8 "Template Analysis and its Application to Natural Language Processing" in Information Storage and Retrieval, Report No, ISR-16, Department of Computer Science, Cornell University, 1969 Wellisch, H. "A Flow Chart for Indexing with a Thesaurus", JP ASIS, 23, 1972, 185-194 Wilks Wilks, Y,A0 Grammar, Meaning, and the Machine Analysis of Language, London: Routledge, 1971 Wilks, Y.A. "The Stanford Machine Translation Project" in Natural Language Processing (Ed. Rustin), New York: Algorithmics Press, 1973 J.Williams Williams, J„H. "Computer Classification of Documents" in Mechanised Information Storage, Retrieval and Dissemination (Ed* Samuelson), Amsterdam: North-Holland, 1968 Williams, J.H. and Perriens, M.P. "Automatic Full Text Indexing and Searching Systems", IBM Federal Systems Center, Gaithersburg, Md., 1968 Williams, JoH, "Functions of a Man-Machine Interactive Information Retrieval System", J. ASIS, 22, 1971, 311-317 Williams, M.E. "Experiences of IIT Research Institute in Operating a Computerised Retrieval System for Searching a variety of Data Bases", I. S. & R., 8, 1972, 57-78 Winograd, T. Understanding Natural Language, Edinburgh: Edinburgh University Press, 1972 Wolff-Terroine, M. and Rimbert, D. "Computer-Aided Automatic Generation of a Structured Documentary Language: Preliminary Study". J. D o c , 27, 1971, 111-124 Woods Woods, W.A. "Transition Network Grammars for Natural Language Analysis", Communications of the ACM, 13, 1970, 591-606 Woods, W.Ao, Kaplan, R. and Nash-Webber, B. The Lunar Sciences Natural Language Information System, Final Report, Bolt, Beranek and Newman, Inc., Cambridge, Mass., 1972 Woods, W.A. "Progress in Natural Language Understanding - An Application to Lunar Geology", AFIPS Conference Proceedings, Vol.42, 1973, 441-450 B.ll