LMSC-D246461 Part II

Part II DESCRIBING AND ABSTRACTING PICTORIAL DATA

LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSOD246461 P a r t II

Section 1 INTRODUCTION

To index and a b s t r a c t pictorial data, much of the information concerning m e t r i c information must be discarded and only the objects, their p r o p e r t i e s , their r e l a t i o n ships to other objects, and their locations a r e retained. Then, by using only a limited Thus, one important number of object n a m e s , modifiers, and r e l a t i o n s , a wide c l a s s of pictures can be r e p r e s e n t e d for future r e t r i e v a l in response to g e n e r a l q u e r i e s . purpose of picture description i s that of generalization, and in our investigation the r e t r i e v a l p r o c e s s consists of trying to match the query s t r u c t u r e to general description structures. It is possible to approach the study of pictorial data in either a formal or a nonformal manner. manner. That i s , one can attempt to p r e s e n t the descriptions in a formal, rule-bound, Much r e c e n t work h a s tended to be formal in nature (Refs. 1, 2) and h a s In our study, we have and analytic manner, o r one can approach the problem in an analogical, intuitive produced disappointingly few r e s u l t s relevant to our i n t e r e s t s . language in u s e . Over the p a s t s e v e r a l y e a r s , we have examined the variety of ways in which human o b s e r v e r s d e s c r i b e and a b s t r a c t pictorial s t r u c t u r e s . by human o b s e r v e r s were examined. s t r u c t u r e of such d e s c r i p t i o n s . After a review of the field (Ref. 3), we c a r r i e d out experiments in which descriptions of pictorial data p r e p a r e d These e x p e r i m e n t s , r e p o r t e d in Ref. 4, gave u s a data b a s e of descriptive m a t e r i a l and allowed u s to examine the terminology and In the first p a r t of the p r e s e n t contract, we examined the problem of description of e a r t h r e s o u r c e s imagery* and indicated in Ref. 5 how a "dynamic 11 data entry s y s t e m , i . e . , one which allows for the entry of m o r e information concerning a given picture obtained from later o b s e r v e r s , could be used to obtain an

taken a nonformal approach, studying pictorial description from the point of view of

*See Table 1-1 for typical descriptions of the various e a r t h r e s o u r c e s disciplines. 1-1
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461 Part II

Table 1-1 TYPICAL DESCRIPTIONS FOR THE VARIOUS EARTH RESOURCES DISCIPLINES
Geography The Nile River and its distributaries within the delta appear as sinuous buff-colored ribbons. The main stream l i e s against the east (right) side of the valley to the south. The Rosetta Distributary is seen winding along the west margin of the delta, while the Damietta Distributary roughly bisects the irrigated area. Meanders and islands are everywhere in evidence. The major canal system appears as a s e r i e s of straight buff-colored lines radiating outward from the settlement nodes. Such straightline segments are nearly always canals rather than roads. Agriculture Infrared photo taken from an airplane at Weslaco,Texas, shows crop vigor, growth, and soil salinity: (1) healthy cotton, (2) unhealthy cotton, (3) bare soil, (4) pig weeds in wet area, (5) pig weeds above short sorghum, (6) dry topsoil between rows of sorghum, and (7) bare soil between sorghum in moist a r e a s . Forestry Figure 13 i s an Ektachrome infrared picture taken by the Apollo 9 astronauts of the test site near Vicksburgh, M i s s i s s i p p i . . . . The dark blotches within this area are uncut patches of hardwood, and a large area of hardwood forest can be seen toward the lower right corner of the photo. Although the seasonal state of deciduous forest affects the appearance of such areas, little difficulty is encountered in distinguishing between cultivated land and forests in space photography. Geology (A Gemini 4 photograph of southwest Saudi Arabia.) Most of the upper half of the photograph is occupied by a range of rugged mountains composed of Igneous rocks. Toward the base is an expanse of sedimentary rocks that dip very gently toward the upper right of the photograph. The distinction between igneous and sedimentary rocks is made by noting the presence of stratification, or layering, in sedimentary rocks. These two units appear to be separated by a major fault. Hydrology On the far side of the Tigris River shown in Fig. 22 is one end of a large lake that, judging from the pattern of dark tones around it, varies in size and is now at a low stage. Between the lake and the river are distributaries — subsidiary streams that carry water away from the main stream and which branch downstream. Such streams deposit large amounts of fine-grained sediment. Oceanography (A Gemini photo of Campeche, Yucatan.) Some of the prominent features here are longshore and offshore currents and a sandbar development. An easterly longshore current i s made visible by the sediments discharged into the Gulf of Mexico's clear water by a river hnmen lately to the left of the photo margin. The sediments are light-colored marls, silts, and sands. To the right of the laguna, the sandbar development is clearly visible. Cartography The valley of the Rio Grand cuts a c r o s s the picture from the upper left to the bottom center. The Franklin and Organ Mountains are near the center. Below them are El Paso, Texas, and Jaurez, Mexico. These cities are in a strategic location in a pass in the range of mountains. There are irrigated areas on both sides of the international borderline. The Rio Grande Valley i s slightly depressed and our highways pass through El Paso.

1-2
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461 Part II

"encyclopedic" data base. The use of a relational net for representing the content of picture description was described, and methods for merging and searching such structures were indicated. A summary of this paper is given in Appendix A. The second portion of this contract period was devoted to the study of representations of the content of descriptions of terrain photography written in natural language. This report is devoted to that problem. There are three basic problem areas in representing the content of a description expressed in natural language: (1) The problem of finding canonical forms, i . e . , how to represent paraphrases of the same concept (2) How to translate from natural language to this representation automatically (2) How to organize the data base so as to keep the representations simple, and still capture the full conceptual intent of the description • The representation problem. Given a set of sentences such a s : Many wharves line the river. On the sides of the river are a lot of wharves. Along the river are a number of wharves. There are many variations of these sentences using "docks, M "river, " and "many" in place of "wharves, " "stream, " and "lot of, " respectively, and it is important that these variations be given the same conceptual representation. The question of selection of the primitive terms used in the conceptual representation arises, e . g . , should "side of" be considered a primitive, or should it be represented as "on the outside edge of an object, "using the primitives "on, " "outside, " and "edge. " Finally, the conceptual representation must indicate the semantic role played by the primitives. • The translation problem. The problem of translating from natural language expressions to a conceptual representation is very difficult. Because there are often many allowable parsings of a given sentence if only syntactic criteria are used, it is

1-3
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E A I K R A F T C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461 Part II

necessary to use additional aids. As indicated in Part I, section 2 . 1 , one possible tool that can be used is the word government table, developed earlier in this study by Robison (Ref. 6), a dictionary of words and allowable adjacent syntactic structures for each word meaning. These associated syntactic structures pinpoint a given word's semantic meaning, and can therefore be used as an aid in deriving the conceptual representation of the sentence. • Data base organization. If we have the description, "an airport is to the left of the freeway, " and represent this as left of (freeway, airport), then a query such as n what is to the right of the airport?" cannot be answered unless the data base has stored the property of the relationship, "left of (a,b) implies right of (b, a). " Similarly, the system must contain axioms which indicate that if an object is inside another object, then it cannot also be outside that object, and that if an object is far from an object, then it is outside that object. If such properties of relations are not included in the system as general rules, then either the representation of each description becomes encumbered with a large number of relationships, or, if such relationships are not present in the representation, then the ability to respond to queries is diminished.

1-4
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSOD246461 Part II

Section 2 REPRESENTATION OF MEANING

A parsing of a sentence indicates how the words interact, and which words or groups of words are related on various levels of aggregation. the words or word groupings. However, to represent the content or meaning of a sentence, one must indicate the conceptual roles played by For example, a representation of meaning should indicate whether a term relates to another so as to indicate quantity (Mthree lakes M ), location (Mat the river t f ), or attribute of an object ("waterway has docks 1 '), etc. It is therefore important in any investigation ot representation of meaning to identify these c l a s s e s , to study natural language expressions representative of their use, and to determine effective techniques for representing content as a function of these c l a s s e s . 2.1 CONCEPTUAL CLASSES FOR PICTURE DESCRIPTION Consider the description of an aerial photograph given in Table 2 - 1 . In the table below the description, we have indicated the n concept kernels 11 — phrases which capture a basic idea or concept. On the right, we have noted the general concept class (or c l a s s e s ) of each kernel. If we represent the meaning of the description by a network in which terms are shown as nodes and the relationships by links, then we obtain a diagram such as is shown in Fig. 2 - 1 . In that figure, the concept c l a s s e s have been used to label the links with the conceptual role played by that link. Such conceptual c l a s s e s serve several r o l e s . First of all, they help to organize the

study of terms and expressions used in description, since we can examine terms and expressions representative of each c l a s s , rather than having to consider all terms and expressions. Second, they can be used in the conceptual net to indicate the semantic role played by relationships and thus simplify the search and analysis tasks when extracting information from the net.

2-1
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSOD246461 Part II

Table 2-1 BASIC CONCEPT CLASSES OF A DESCRIPTION Description by human observer It is an aerial photograph, whose most arresting feature is a waterway, possibly an estuary or part of a harbor, with many ship docks. The waterway stretches across the upper part of the photograph with a single stream on the left breaking into three arms at not quite midphoto. The upper arm divides again into inlets cut off from view; at the top, the middle arm is a small, straight-sided, closed inlet or docking area in a peninsula or island; the lower arm passes out of the picture on the right, possibly dividing again at the right edge. Across the waterway just to the left of the division is a bridge. Concept Kernels of the Description It is an aerial photograph Most arresting feature is a waterway Waterway is possibly an estuary or part of a harbor Waterway has many ship docks Waterway stretches across upper part of the photograph Waterway breaks into three arms Break is at not quite midphoto The upper arm divides again into inlets [Inlets are cut off from view At the top of the middle arm is a docking area or inlet Inlet/docking area is straight-sided and closed Inlet/docking area is in a peninsula or island The lower arm passes out of the picture on the right The lower arm possibly divides again at the right side j Across the waterway is a bridge J Bridge is located just to the left of the division of the waterway Concept Classes Photograph as physical object Attribute of object Classification of object Set membership (has) Operative (stretches across) Operative (partitions) Location of object Operative (partitions) Existence Existence location Attribute of object Location of object Existence (viewability) Operative (partitions) Operative (crosees) Location of object

2-2
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461 Partn

§
U

o
CA

a i
o

s s

s 1
CD

1
CM

bio

2-3
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461 P a r t II

Finally, many v e r b s a r e associated with certain conceptual c l a s s e s and allowable word forms, and such c l a s s e s lead to constraints on the s t r u c t u r e of the sentence. These constraints can be analyzed and tabulated for use as a guide in performing both the syntactic and semantic a n a l y s i s . Recently, s e v e r a l approaches to linguistic analysis based on concept c l a s s e s or " c a s e s " have been presented by investigators such as F i l l m o r e , Schank, and T e s l e r . of this work in Appendix A. In examining descriptions of a e r i a l photographs, we have chosen the following s e t of concept c l a s s e s for r e p r e s e n t i n g meaning of picture description. • 1-8 Case r e l a t i o n s . Relations such as (1) location, (2) localizing,(3) qualifying (4) quantifying, (6) operative, (7) use, (8) existence, which a r e often strongly related to the v e r b of the sentence. • • • • • • 9 Attributes. Attributes of an object, such as color, texture, shape, s i z e . Many of the attributes have a value, e . g . , "red, " "rough, " e t c . 10-Comparison. Comparisons can be made on the values of attributes, e . g . , "redder than, " "rougher than, " "bigger t h a n . " 11-Logical linking. Linking of concepts using logical relationships such as "and, " "or, " "not, " "if object. 13-Doubt, approximation, or uncertainty. or uncertainty on the p a r t of the o b s e r v e r . 14-Picture properties. Expressions which deal with the physical p r o p e r t i e s of the picture, such as size, resolution, and picture quality. Some of t h e s e conceptual c l a s s e s a r e described m o r e completely below. Indications of doubt, approximation, then" 12-Set m e m b e r s h i p . Indication that an object is a subset or p a r t of another We have examined c a s e g r a m m a r s which identify the underlying syntactic-semantic relationships, and give a brief s u m m a r y

2-4
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSOD246461

Part n

2 . 1 . 1 Case Relations The first eight conceptual c l a s s e s grouped together under the term n case relations' 1 are listed in Table 2-2 each c a s e . which shows the role played by each case and examples of It is expected that the list will be modified as the analysis proceeds.

Because location tends to figure strongly in observer descriptions of pictures, we investigated expressions for location in some detail. In Table 2-3 (a tabulation of expressions used to denote location), expressions appearing in the same box have roughly the same meaning, while those appearing in adjacent columns are either antonymns, or express somewhat related ideas. suitable general name can be found. Similar concepts have been grouped together, and the column on the left indicates the name of the grouping whenever a The question of suitable primitives for these expressions will be discussed in section 2 . 2 . 2 . 2 . 1 . 2 The Remaining Conceptual Classes The remaining six conceptual c l a s s e s are described briefly below. • Attributes. Various attributes that appear in pictorial description are indicated Each attribute has a value, e . g . , the value of "color" may be "red, "

in Table 2-4.

and often the values of the attributes can be compared, as described below. • Comparison. Comparison of attribute values can be indicated in the conceptual Suppose we wish to indicate, "object A is longer than object B . "

net as follows. We use the net,

ATTRIBUTE LENGTH

9 1
I

(B) ATTRIBUTE

1

LENGTH

^

I
— Value (LENGTH)

Value (LENGTH) - ©

to indicate that the value of the LENGTH attribute of A is bigger than that of B. 2-5
LOCKHEED PALO ALTO RESEARCH LABORATORY
t O C K H E f O
A OPOUP

M I S S H E S
OF

&

S P A C E
AIUCPAFT

C O M P A N Y
C O P P O P A T I O N

D I V I S I O N

I O C K H E E D

LMSC-D246461 Part II

Table 2-2 CASE RELATIONS USED IN PICTURE DESCRIPTION

Case 1. Location/extent

Role Played Location of object with respect to frame of picture, or with respect to another object. Also extent of object Portions of objects (can be thought of as location with respect to object itself) Which of an object is meant

Examples X is near Y, X is west of Y (see Table 2-4)

2.

Localizing

Top of X, edge of X middle of X The upper arm (to distinguish this arm from other arms) Three arms, few, many X is a Y X appears to be a Y Y seems to be a Y X is similar to a Y X resembles a Y X is classified as a Y X is related to a Y X crosses Y X cuts Y X disrupts Y X intertwines Y X invades Y X divides Y X stretches across Y X developed for Y X used by Y There is an X an X can be seen an X is visible an X can be found an X is observed an X will be noted an X appears

3.

Qualifying

1 4. 5.

Quantifying Classificational

How much or how many An entity in the photo is identified as to class or name

6.

Operative

An entity does something to another entity (usually intrusion, crossing, or cutting)

7. 8.

Use or application Existence

Purpose of object An entity can be seen

2-6
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461

Part n

Table 2-3 EXPRESSIONS FOR LOCATION
Concept Distance to X X X X X X X X is is is is is is is is Expressions Used X i s far from Y at Y in the vicinity of Y X is away from Y X is distant from by Y with Y close to Y near Y toward Y in place Y X X X X is is is is under Y underneath Y below Y ; beneath Y X i s (distance) from Y

X i s on Y X i s above Y X i s over Y X i s in Y X i s inside Y X i s within Y X i s here X i s between Y and Z X i s around Y X i s alone Y X i s among Y Relative position X i s to the left of Y X i s East of etc. X i s in front of Y Orientation

X i s off Y

X i s out of Y X i s outside of Y X i s here

X i s along Y

X i s everywhere

X i s to the right of Y X i s West of etc. X i s in back of Y X i s behind Y

to the side of

X i s perpendicular to Y X i s parallel to Y X i s a diagonal X runs north (south....)

X makes an angle of Q with Y

2-7
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E A I K I A F T C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461 Part II

Table 2-4 ATTRIBUTES OF OBJECTS Attributes Shape Values of Attributes curved straight round big, little short, long, 5 in. big, small, one acre 1 red, blue light, dark bright, dull grainy, sandy, mottled smooth sprinkled with scattering of clumps of Comparatives of Values more curved, curvier, straighter rounder same shape as greater than, smaller than, equal redder than, bluer than, same color as lighter than, darker than, same grey level as, brighter, duller rougher than, grainier than, rough as smoother than smooth as

Size Length Area Color Grey Level

Texture

•

Logical linking. Objects are often linked using logical relationships such as Some of the expressions used are shown below. Symbol Used * +
t>j

M

and, n

"or, " Mnot, M "since x, then y, " etc. Such linking can be indicated using the standard logical notation.

Logic Expression and or not if X then Y

Natural Language Form roads and mountains oil or water does not cross X shows that Y X indicates that Y X confirms that Y X gives evidence that Y judging from X, then Y because of X, then Y

X —Y

2-8
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461 Part II

•

Set membership. Indications that an entity is a subset of another entity are given

by such natural language expressions a s : A is part of B A i s an element of B A belongs to B • Doubt/uncertainty/approximation. It is important that the links of the conceptual

net retain an indication of doubt, uncertainty, or approximation concerning observations, s i z e , and quantity. mately,
tf

Thus, a value of an attribute should be labeled as •'approxi-

and that of an observation u possibly, " when so indicated by the observer.

It may be desirable to use a scale of approximation or of possibility, but this should offer no difficulty. • Picture properties. Because of the importance of such picture properties as clarity,

resolution, scale, e t c . , we have separated this topic area out as a conceptual c l a s s . 2.2 PICTURE PRIMITIVES

The terms appearing in the relational net should be selections from a basic set of primitives to enable matching to be performed between queries in canonical form and the data base. below. 2 . 2 . 1 Attribute Primitives Some of the attributes concerning size, quantity, and grey level (given previously in Table 2-3) permit representation in terms of a small s e t of primitives. For example, if one defines BIG as a general primitive denoting a large value of an attribute, then Primitives appropriate for the various conceptual c l a s s e s are described

2-9
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A O l O U f M I S S I L E S OF & S P A C E A I I C I A M C O M P A N Y CORPORATION D I V I S I O N l O C K H I f D

LMSC-D246461 Pari II

we can define
QUANTITY: few a lot, many = ~ BIG = BIG LENGTH: short long not long and not short GREY LEVEL: » ~ BIG = BIG light dark greyish, not dark and not light = ~ BIG - BIG = ~ (BIG* ~ BIG) = ~ BIG = BIG = ~ (BIG* ~ BIG)

several, some - ~ (BIG* - B I G ) SIZE: little, small large, big

moderate sized = ~ (BIG*-BIG)

However, the primitives for shape, direction, color, and texture are not as straightforward. In the case of shape, for example, one could define as primitives a few selected geometric shapes. Human description of shape, however, often uses the names of objects to indicate shape, e . g . , "S-shaped, " "hook-shaped, " and it is not clear whether any set of primitives of reasonable size can be found to cover the spectrum of such descriptions. For color, one can use the primary colors, indicating the other colors in terms of primaries, e . g . , "orange" would be lf red/yellow, " and shades could be indicated using BIG. Thus, an orange that was on the reddish side could be indicated "BIG-red/ yellow. M Texture is a difficult class to deal with in terms of a small set of primitives, and it is not clear as to what an acceptable set of such primitives might be. 2.2.2 Location Primitives ,\ tentative set of 15 to 20 location primitives chosen from Table 2-3 is currently being evaluated. The number of such location primitives can be reduced by

2-10
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461

Partn

defining a s e t of primitives based on the idea of an object being inside or outside a suitably defined space o r region. as follows: space(Y) nearspace(Y) betweenspace(Y, Z) = the space defined by the boundaries of object Y = the space near object Y, where "near 11 would depend on the particular problem a r e a » the space between Y and Z amongspace(Y, Z, . . . ) = tho space in a region of a s e t of objects Each of the spaces or regions i s assumed to have an edge, a s does an object. One then can define a s primitives "inside, " "outside, " and "on, " having their usual meanings. as follows: Expression X is inside Y X i s far from Y X is along Y X i s between y and z In T e r m s of Primitives inside(X, space(Y)) outside(X, nearspace(Y)) on(X, edge (nearspace(Y)) P a r a p h r a s e of the Notation X i s inside the space of Y X i s outside the nearspace of Y X is on the edge of the n e a r s p a c e of Y inside(X, betweenspace(Y, Z)) X i s inside the betweenspace of Y and Z We can then define many of the location t e r m s using these primitives T e s l e r (Ref. 8) has used such a s e t of primitives,

Localizing t e r m s such a s "top of," "bottom of, " "right side of, " which indicate portions of an object a r e thought of a s defining a region. while i t s inside i s " b e l o w . " 2,2.3 Localizing P r i m i t i v e s Thus, the outside of "top" is "above, "

Localizing t e r m s such a s "top, " "bottom, " and " s i d e " will be expressed in t e r m s of the following p r i m i t i v e s .

2-11
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461 Part II

Primitives top right | front edge

Examples of Use of Primitives bottom • ~ top; center = 1/2 (top) left = back = edge ~ right ~ front

J possible combining into one term portion '

2.2.4 Qualifying Primitives Qualifying terms, such as Mthe upper arm, " are used in natural language expressions to denote which of several objects is being referred to. In terms of the conceptual net, they indicate the proper link connections, but do not appear as a high level identifier of a link. 2.2.5 Classificational Primitives The classificational primitives depend on the subject area or discipline for which the description is being prepared. A set of such primitives would therefore have to be prepared for each such discipline. 2.2.6 Operative Primitives To deal with terms such as "cuts, " "crosses, " etc., our initial selection is one term which indicates a crossing over, one term which indicates a cutting off, and one term which indicates entry into, as shown below: cuts across • crosses, spans, divides, partitions cuts off = disrupts cuts into » invades, enters

2-12
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSOD246461 Part II

2.3 STRUCTURING THE DATA BASE We have considered several strategies for structuring the data base s o as to allow efficient search of the pictorial descriptions.
M

Two approaches are the use of the

metanet, " and the use of relationship properties.

2 . 3 . 1 The Metanet The metanet i s a high-level network which uses only conceptual c l a s s e s and not the relationships themselves to label links. In addition, only the T?upperM levels of the original conceptual net are retained. structure of the actual conceptual net. 2 . 3 . 2 Relationship Properties To keep the number of links appearing in a conceptual net to a minimum, it i s desirable to make available to the retrieval system certain "world knowledge M concerning relationship properties. For example, instead of using the net Search in response to a query i s first to be made on the ordered s e t of metanets, each of which indicates the general theme or

* is to the right of -v

©
right of (a,b) = left of (b, a).

©

M s to the left oi~S it i s preferrable to use the net (a)— i s to the right of —(b), and to store the rule:

Additional properties of relationships are required because mismatches can occur between query and data base even when relationships are reduced to a primitive form. For example, "at" and "near" cannot be expressed by the same primitive, yet something which i s "at11 a location is also "near" that location, and therefore the

2-13
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E A I K R A F T C O M P A N Y CORPORATION D I V I S I O N lOCKHEEO

LMSC-D24G461 Part H

property, at(m,n) implies near(m, n) should be part of the system. the aid of Ref. 9.

An example of

properties of relations concerned with location is given in Table 2-5, derived with

If an extensive body of world knowledge is to be provided to the system, some efficient method of searching this collection of facts must be provided. That i s , the system Although must somehow locate relationship properties that are relevant to a query without performing an exhaustive search of the collection of its world knowledge. it may be possible to develop a suitable organization and indexing scheme for this collection, the alternative of portraying this information using a two-dimensional map s e e m s more attractive. "experiments" in the space. Using this approach, various concepts are defined For example, if the relationship,
M

individually in this space, and the interrelationships are determined by performing X is between Y and Z" is depicted in this space, (where Y and Z are parallel lines and X is a point), then, as X approaches Z so that MX is nearer to Z, M the system can note experimentally that MX is farther from Y . "

We plan to explore the two-dimensional portrayal of two-dimensional relationships during the next contract period.

2-14
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461 P a r t II

Table 2-5 PROPERTIES OF RELATIONS CONCERNED WITH LOCATIONS Reciprocal relations Near(m, n) F a r from(m, n) Beside(m, n) = Near(n, m) = F a r from(n, m) = Beside(n, m)

Opposite relationships Upon(m, n) On(m, n) Above (m, n) = Under ( n, m) = Under( n , m ) = Below ( n,m)

Mutually exclusive relationships Inside(m, n) Near(m, n) Below(m, n) Below (m, n) Above (m, n) At(m, n) Inside(m, n) Implications Near(m, n) F a r from (m, n) At(m, n) Upon(m, n) Upon(m, n) Under (m, n) implies implies implies implies implies implies Outside(m, n) Outside (m, n) Near(m, n) At(m, n) Above (m, n) Below (m, n) excludes excludes excludes excludes excludes excludes excludes Outside(m, n) F a r from(m, n) Above (m, n) Beside(m, n) Beside(m, n) Between(k, m, n ) W Between(k, m, n)

( a ) i . e . , k cannot be between m and n.

2-15
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A G I G U P M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461

Partn

Section 3 NATURAL LANGUAGE ASPECTS OF CONCEPTUAL MAPPING

The previous section dealt with problems of representation of meaning; this section deals with problems in determination of meaning from natural language expressions. 3 . 1 SEMANTIC AMBIGUITY To indicate the problems of ambiguity in picture description, let us examine three descriptions from Table 1-1; often sentences that offer no problem to the human are quite difficult for an automated process to handle. Consider the following sentence: (1) The Nile River and its tributaries within the delta appear a s sinuous buffcolored ribbons. Note that "appear11 can be appear± - arc represented a s or appear^ = come to sight. The program must, from a government table, identify the appear a s form a s indicating appear! , and the parsing program must then be able to identify the two subjects the Nile River, and i t s tributaries and that within the delta i s a prepositional phrase modifying tributaries. If we use the primitive terms, a c inside of consisting of b are represented a s e i s visible in c approximately in

then the conceptual net for sentence 1 would be mapped a s , Nile River ^ s~\ jancflare represented a s — sinuous buff-colored ribbons * tributaries ^^^ ( a } inside of
W

1
delta

3-1
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GIIOUP M I S S I L E S OF & S P A C E A I K I A F T C O M P A N Y CORPORATION D I V I S I O N l O C K H f f O

LMSC-D246461 Part II

(2) Such straightline segments are nearly always canals rather than roads. Here we are faced with the ambiguity of a r e . = identity, and are = represent. The program would have to identify the are meaning and the rather than would have to be identified as not to obtain the representation, straightline segments « are represented as — canals — straightline segments — not (are represented as) — roads (3) The dark blotches within this area are uncut patches of hardwood, and a large area of hardwood forest can be seen toward the lower right corner of the photo. The can be seen has the alternative meanings, can be seen. = is visible, and can be seen^ = are represented a s . The program would have to handle the ambiguity of both a r e , and can be seen to obtain the conceptual net,

©
dark blotches *(a) inside of area are represented as uncut patches (d) consisting of hardwood

©
large area (d) consisting of is visible in '

^

1
(c) approximately in lower right corner (a) inside of

i
hardwood forest

i
photo

The importance of the government tables in resolving ambiguity can be seen from this brief examination of mapping from natural language to concepts. (It should also be noted that not all ambiguities are resolved by such tables.) A word government table

3-2
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSOD246461

Part n

suitable for pictorial description is being prepared by extracting and expanding upon the entries from the Robison tables • An example of the entries for the letter a. as extracted from the Robison tables is shown in Table 3 - 1 . The tables to be constructed will be in the form described in Part I, section 2.2 (which includes word meanings), and is expected to have l e s s than 1000 entries.

3 . 2 PARSING PICTORIAL DESCRIPTIONS The PHRASE parsing program of Lois Earl was used to analyze four descriptions as follows: (1) an earth resources geological description, (2) a description of an aerial photography by a lay person, (3) description of a surgical scene by a surgeon, and (4) a botanical description. The resultant parsing are shown in Fig. 3 - 1 . It will Two of these errors be noted that there were nine sentences in the descriptions, and there were seven phrase-constructions whose function was incorrectly labeled. due to incorrect interpretation of conjunctions. These results were obtained using both Level I and Level II parsing, as defined in Section 1. needed. The Level II parsing is necessary for the identification of word government patterns in text, while for full semantic analysis, all four levels of parsing will be During the next contract period, the Level III parser will be available for parsing these descriptions, and it will be interesting to see what improvement in parsing results. were due to incorrect part-of-speech determination of technical words, and two were

3-3
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A CROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461 Part II

Table 3-1 ENTRIES FOR LETTER "A" FROM GOVERNMENT TABLES PERTINENT TO PICTURE DESCRIPTION

absence absent | access accessible I accompany add

1 2

n n vl2 aj n aj

of S/from S of S/in S I*/from S from S of S/to S/by S to S/by S |

alignment 1 amount angle

n n vt vt n n aj av vt vt n n 1 1 2 2 n n n n vi n 1 1 2 vi vi vi n vi 1 1 1 1 2 2 n n n n n n n n n

of S/wlth S of S/in S S/at S S/toward S of S/with S between S/and S to S from S S/to S S/onto S to S of S of S/into S of S/to S of S/between S/and S/ according to S of S/among S/according to S to S of S/to S from S out of S at S of S at S/by S of S/wlth S between S/and S of S/to S/with S of S/to S/by S between S/and S of S/for S of S of S/with S of S/by S

anterior apart append appendage apportionment

1 2 1 1 2 3

vt vt vt vt vi vl aj vt n vt vt vt vt vt vt n n n n n n vt vt vt aj vi vi vt

S/to S S/with S S/onto S S/to S to S up to S to S S/to S to S S/toS S/toward S S/into S S/on S/from S S/upon S S/from S/to S/over S of S/to S of S/toward S of S/lnto S of S/on S/from S of S/u)>on S of S/to S/ovor S S/U) S S/onto S S/u|M>n S of S/by S/in S for S at S S/with S

1 adjacent i adjoin | adjunct 1 advance

approach arise

arm arrive association attachment

affix

1 ahead 1 aim align

attribute augmentation

3-4
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461 Part II
Description No 1, KM i lit HCHOUII IH, •cologjcul DeHi Ilotion I'HI' NAP The muni oltvioUM leuture NAI» NAP Vlbll.l. I, the s p a c e d all m r NAI'
CJ

the lar^e NAP

VIII'

r~ drilling sand durua that completely cover tin Udrock. PHP I CJ NAP \ NAP u

and bruMd \x dlim nU

or bedrock sreaa

— \

PHP NKPP I \ I NAP NKP NKP NAP NA-VH Vlll I \ I \ / \ I \ v«ry thin wuati nl recent u> quaternary alluvium AV PHI

NAP

CJ

NAP

NAP

VHP

AJ

InvelberKN and •mall ruck outcrops In the pedlnn ntn are

numerous.

Description No. 2, Description ol Aerial Holograph by I .ay Observer PH P NAP The picture VHP lit NAP NAP PV-PP VH-AU PK AH Ly

u color aerial photograph of u land area Invaded

CJ VBP NAP r~\ i \ i » or three-pronged waterway PHP NAP Wharves VBP line NAP NAP NAP AJ-AV PHP NAP

u

lion shaped

the sides ol Um waterway , a bridge, probably lor auto trallle PHP PHP VHP misses NAP NAP

AV I'll CJ
iHjt

AJ-AV possibly

NAP

lor rail trallle

the handle ol the Irllon

Inscription No

J. Description ol a Surgical Scene by Medical Expert PHP / \ N In length PHP PHI I NAP NAP NAP ~\ f~ 1M roughly (uHltorin In shape; tla VHP

NAP An appendix.

NAP 6 2 centimeters

NAP midpoint

VBP NAP \ r-\ i \ la I 6 cm. NAP there

r
AV PH

NAP dime

r I I I -\

PHP NAP At this point, PHP NAP

VHP la

NAP a tan excres< ence extending 1. 3 centimeters PHP \ I NAP VHP NAP In tan NAP In contrast \ I

PHP above 1 NAP to the blue-gray

PHP

PHP

~v r

NAP

the level of the appendiceal wall that PHP NAP PHP NAP

of the organ as a whole.
Description No. 4. BoUnlcal Description

NAP

VBP

NAP

VBP

NAP

NAP

CJ

I
NAP obsolete stipules.

\ /—\

Moilugo are low glabrous much-branched annuals with whorled leaves and

NAP The flowers

VBP

NAP on slender pedicels.

-\ m axillary, are r

I \t—\ I \ There are five scarloua margined sepals

Fig. 3-1 Description Parsings Using BPHRAS 3-5
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D
A GROUP

M I S S I L E S
OF

&

S P A C E
AIRCRAFT

C O M P A N Y
CORPORATION

D I V I S I O N

LOCKHEED

LMSC-D246461

Partn

Section 4 REFERENCES

A. C. Shaw, " P a r s i n g of Graph-Representable P i c t u r e s , " ACM Journal, Vol. 17, No. 3, 1970 M. B . Clowes, "Pictorial Relationship — A Syntactic Approach, " Machine Intelligence 4, B . Meltzer, and D. Michie, e d s . , New American Elsevier Publishing C o . , I n c . , 1969, N. Y. O. Firschein and M. A. F i s c h l e r , "Describing and Abstracting Pictorial Structures, P a t t e r n Recognition Journal, Vol. 3, No. 4, Nov 1971 O. F i r s c h e i n and M. A. F i s c h l e r , Pictorial Data,
f! M n

A Study in Descriptive Representation of

Second Int. Joint Conf. on Artificial Intelligence, Sep 1971,

Iiondon, England O. F i r s c h e i n and M. A. Fischler, "Descriptive Representation of Remotely Sensed Image Data,
M

1971 IEEE Systems, Man and Cybernetics Group, Annual

Symposium Record, IEEE Catalog No. 71 C46-SMC, Oct 1971, Institute of E l e c t r i c a l and Electronic Engineers, N. Y. H. R. Robison, "Computer-Detectable Semantic Structures, " Information Storage and Retrieval, Vol. 6, Pergamon P r e s s , 1970, Oxford, New York, London, and P a r i s L. T e s l e r , "New Approaches to Conceptual Dependancy Analysis, " in Spinoza I I : Conceptual C a s e - B a s e d Natural Language Analysis, by R . C. Schank, L . T e s l e r , and S. Weber, Stanford Artificial Intelligence Project Memo AIM-109, J a n 1970 E. Sandewall, "Representing Natural Language Information in Predicate Calculus, " in Machine-Intelligence 6, B . Meltzer and D. Michie, e d s . American Elsevier Publishing C o . , I n c . , New York

4-1
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSOD246461

Partn

9.

C. J. Fillmore,

M

The Case for Case, M in Universals in Linguistic Theory,

E. Bach and R. T, Harms, Holt, Rinehart and Winston, Inc., Chicago, 1968 10. R. F. Simmons, "Some Relations Between Predicate Calculus and Semantic Net Representations in Discourse, " Second Int. Joint Conf. on Artificial Intelligence, Sep 1971, London, England 11. R. C. Schank, L. Tesler, and S. Weber, "Spinoza II: Conceptual Case-Based Natural Language Analysis, M Stanford Artificial Intelligence Project, Memo AIM-109, Jan 1970 12. R. C. Schank, "Finding the Conceptual Content and Intention in an Utterance in Natural Language Conversation, " Second Int. Joint Conf. on Artificial Intelligence, Sep 1971, London, England

LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461 P a r t II

Appendix A THE USE OF CASE STRUCTURES IN SEMANTIC MAPPING

The "case 1 1 notions c o m p r i s e a s e t of universal concepts which identify certain types of judgments human beings a r e capable of making about the events that a r e going on around t h e m ; judgments about such m a t t e r s as who performed an action, who did it happen to, with what was it done, and what was changed. the meaning of a sentence. Fillmore, Ref. 10, views the sentence as consisting of a v e r b and one or m o r e noun p h r a s e s , each associated with the v e r b in a p a r t i c u l a r case r e l a t i o n s h i p . the c a s e s and their definitions a r e given in Table A - l . Some of He gives a slight indication For example, the The p r e s e n t appendix reviews some of the work in the use of case concepts to obtain a representation of

of how one can go from the surface s t r u c t u r e of the sentence to the underlying c a s e s by noting the c h a r a c t e r i s t i c s of the verbs and the prepositions. Agentive preposition is b ^ , the Instrumental preposition is by if t h e r e is no Agentive, otherwise it is with; the Objective and Facitive prepositions a r e typically deleted, and the Dative preposition is typically t o . Simmons and Bruce, Ref. 11, use the Fillmore case concept to obtain an "attributevalue " r e p r e s e n t a t i o n for the meaning of a sentence. the c a s e s a s follows: For example, given the sentence, "John made c h a i r s with tools on October 20th in Austin, " they get a d i a g r a m based on

NPI

NP2

NP3

NP4

NP5

i
John

i
chairs tools A-l

i
Oct 20

i

i
Austin

LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D A GROUP M I S S I L E S OF & S P A C E AIRCRAFT C O M P A N Y CORPORATION D I V I S I O N LOCKHEED

LMSC-D246461

Partn

Table A-l CASES USED BY FILLMORE

Agentive (A). Instigator of the action identified by the verb. Instrumental (I). The inanimate force or object causally involved in the action or state identified by the verb. Dative (D). The case of the animate being affected by the state or action identified by the verb. Factitive (F). The case of the object or being resulting from the action or state identified by the verb. Locative (L). The case which identifies the location or spatial orientation of the state or action identified by the verb. Objective (O). The semantically most neutral case, the case of anything representable by a noun whose role in the action or state identified by the verb is identified by the semantic interpretation of the verb itself (conceivably, the concept should be limited to things which are affected by the action or state identified by the verb).

This diagram can also be represented as a series of triples:
make make make make make NP1 NP2 NP3 NP4 NP5 AGENT OBJECT INSTRUMENT TIME LOCATION TOKEN TOKEN TOKEN TOKEN TOKEN NP1 NP2 NP3 NP4 NP5 John chairs tools Oct 20 Austin

A-2
LOCKHEED PALO ALTO RESEARCH LABORATORY

LMSC-D246461

Part n

Since this s e t of t r i p l e s r e p r e s e n t s the meaning of the sentence, Simmons and Bruce suggest that to answer a question one simply finds the s e t of triples in the data base corresponding to the triples in the question. procedure. Schank et a l . , Refs. 12, 13, use case concepts in their p a r s e r , Spinoza EL these actions. The They do not deal with the problem of p a r a p h r a s e , a problem that keeps such matching from being a straightforward

p a r s e r makes use of relations between conceptual actions and the implications of This enables the conceptual analyzer to discover not only the conceptual The For content of an utterance, but also the intention of that utterance in context.

conceptual dependency analysis which they use is directly related to the case concept, but they use an interesting generalization to handle the problem of p a r a p h r a s e . example, the sentence,
?

'the man took the book, " would have the network

p man < • take

o —

book

R «

•to 1
1

man X

from

where p = actor or p e r f o r m e r , o = object, R is the recipient, and the X denotes the fact that the entity from which the book was taken was not specified in the sentence. However, this network is typical of an entire c l a s s of actions involving transfer of an object from one entity to another, e . g . , "given, general network
n n

send, " "steal, " which have the

Z * = * trans

o R — object « —

The v e r b dictionary used in Spinoza n would indicate that for "give" the " t r a n s " network above is relevant, and that Z = X, i . e . , the person performing the action is the s a m e a s the p e r s o n from whom the object c o m e s .

A-3
LOCKHEED PALO ALTO RESEARCH LABORATORY
L O C K H E E D M I S S I L E S & S P A C E C O M P A N Y

LMSC-D246461

Partn

The cases used by.Schank are shown in Table A-2, and those of Tesler, a coworker, Ref. 8, in Table A-3. Table A-2 CASES USED BY SCHANK Conceptual cases Objective Recipient Instrumental Directive [ Attributive Cases Possession Location-near Containment Prepositions Used (none) to, from with, by to, from, toward Prepositions Used of, with near, at, by, in before in, of Relationship Object of the action Receiver or transmitter Instrument used Direction of action Relationship Has Located Contained

Table A-3 CASES USED BY TESLER Conceptuali zation Class Attributive Classificational Behavioral Motive Operative Mutual Relationship Existence, has attribute, in a state Something which is the same as something else Behavior without change of state, victim, goal, or direction,e.g. rotate, dance Something in motion from one station to another Something active does something to something passive Two or more actors interacting to perform an action, which would not be observed to be performed just by observing one of them, e.g. fight An attitude toward an issue, e . g . , say, claim, know, fear, hear Something makes something happen, either intensionally or unintentionally

Hypothetical Causal

A-4

Appendix B
O. Firschein M. A. Fischler Lockheed Polo Alto Research Laboratory Palo Alto, California 04304

LMSC-D246461 Part II

DESCRIPTIVE REPRESENTATIONS OF REMOTELY SENSED IMAGE DATA*

There is a growing interest in high-altitude imago data obtained from various s e n s o r s for use in earth r e s o u r c e s applications. The lines, patterns, and colors in such imagery reveal what both nature and man have done to the face of the earth. In photos taken from high altitudes, highways appear as networks of fine lines, farm fields produce recognizable patterns, and the areas from which forests have been cleared stand out brightly. Colors are indicative of the vegetation, soil conditions, depth of water, and many other matters of vital concern to mankind. The present paper deals with the nature of the image descriptions used to express the content or meaning of earth resources satellite data, rather than the mechanics of handling such data, such as is given in Ref. 1. Natural Language Descriptions Considerable variation occurs in the scope of natural language description of imagery from the various earth r e s o u r c e s d i s c i p l i n e s . 2 Geology, geography, hydrology, and oceanography tend to more global descriptions, while agriculture, forestry, and cartography tend to focus on particular attributes of the image, often relying on o v e r lays or line drawing representations. In addition, we note that the grammatical structures for describing imaged data tend to be much simpler than conventional samples of literary or spoken language - the verb forms and relationships are s i m p l e r , and ambiguity is e a s i e r to r e s o l v e . Several questions arise in dealing with such d e s criptions. (L) Mow can the descriptions be put into some normal or canonical form to regularize variations In description from observer to observer, and also for use in computer based s y s t e m s ? (2) How can a number of different descriptions of a given image be combined to obtain a more comprehensive description? (3) How can the entities and relations between entities in a description be converted to a small set of basic primitives to simplify the canonical f o r m s ? A basic philosophical question also a r i s e s in d e s i g n ing retrieval s y s t e m s : Should one determine the best query structure for a particular user population and then design a suitable data base structure, or should one use a data base structure having certain desirable properties and design the most appropriate query structure? We are taking the latter c o u r s e . By examining various representations of natural language description, we hope to determine structures that have certain d e sirable properties such as allowing automated structure determination from text, and allowing automatic c o m -

bination of separate descriptions of the s a m e image. Query and retrieval s y s t e m s would then be designed around such representations. Canonical Forms and Description Merging We have examined two types of canonical forms, the descriptor approach in which a set of words or phrases is selected to represent the image, (see our review in 5) and the relational graph approach in which a conceptual representation is obtained by indicating relationships between entities in the picture. If two descriptor representations have been prepared for the same image, a combined descriptor set can be obtained by forming a single combined list of descriptors. If two network representations for the same image are available, the problem is more complex but, abstractly, the networks are combined based on detection of common nodes. In the experiments described in 5 , using a set of subjects having no special technical background, convergence to a merged form was obtained quickly; the relational net remained relatively unchanged when additional descriptions of the same image were c o m bined with it. However, in the c a s e of experts dealing with earth r e s o u r c e s data, an individual with a specialty can introduce a whole new set of links and nodes into the net, and therefore convergence to a particular r e p r e sentation does not guarantee completeness; it merely indicates completeness for a particular set of specialties. The Concept of an Encyclopedic Entry This study is concerned with the problem of whether a representation of a picture could be used to answer general questions about the picture. There would s e e m to be too many diverse questions that could be asked concerning the picture, and too many different requirements in terms of the level of response required to allow such a procedure to be successful. Consider, however, an analogous situation which a r i s e s in the attempt to summarize all human knowledge in such a way that questions can be answered using this summary. Although this task would also s e e m to be overwhelming, collections of knowledge, called "encyclopedias," have been used for many y e a r s , and turn out to be a useful means of a n s w e r ing a diversity of questions. Similarly, it should be possible to develop a question-answering system based on the concept of an encyclopedia for the body of knowledge residing in either a single photograph or a set of photographs. The same problems of level of detail and use of technical terminology a r i s e in the preparation of both an encyclopedia based on knowledge expressed in natural language and

•The work described here was supported partially by the Office of Naval Research Contract N00014-70-C-0239, and partially by the Lockheed Independent Research Program.

Paper presented at Joint National Conference on Major Systems, sponsored by the IEEE and ORSA, Anaheim, California, October 1971.
T*_I

LMSC-D246461

Partn
one tn which knowledge IH expressed In pictorial form. In both cases, one hafl to consider how the material IH to be organized, the nature of the potential user, ami the amount of material to be gathered. Relation of Queries to Descriptions We can gain some insight into the requirements for an encyclopedic entry by examining the types of query that aro likely to occur, it should be kept in mind that an encyclopedia contains material suitable for answering many questions, but not ail questions on a topic. In addition, while several diverse items of information from the encyclopedia may be pulled together to answer a question, there is no guarantee that this can be done. Similarly, for an encyclopedic entry for an image, we should expect to be able to answer many questions, to , combine elements of the entry to answer other queries, and to fail completely for the remaining queries. In our examination of queries concerning earth resources imagery, we have found that the following elements of description are often required. Discontinuity and contrast. An indication of discontinuity and contrast by identifying regions homogenous in such attributes as texture, color, pattern, and temperature (on BR) imagery. Terraine structure identification. Queries will often ask for the existence of flow patterns, surface film patterns, silting and sediment, circulation, thermal structure, man-made features, etc. Change detection. Queries often ask for changes in cultural features, stream flow, reflectivity, crustal structure, snow areas, sand bars, etc. Such questions can be answered if previous and present descriptions of an image are available and contain line drawing extracts or equivalent. Such line drawing extracts can be readily compared and the differences noted. Boundary area, and volume analysis. The boundaries of snow and ice, lakes, tectonic features, crop areas, etc., are often required for area and volume computations. Counting of items. Queries will often ask for a count of the number of items, or some Indication of what percent of the image is taken up with some items. Content in a broad sense. These are queries that concern the global aspects of the image, such as: is the area predominately rural? ; what are the general characteristics of the mountain ranges?; what are the transportation networks?; describe the shoal formations. The natural language descriptions are most appropriate for the last type of query. The Nature of an Encyclopedic Entry An enc>clopedic entry for an Image capable of dealing with the above queries would consist of one or more of the following content-indicating elements. Image citation data. Identification information concerning the source of the Imagery, the equipment used, time, date, geographical location, image quidlty, etc. The image citation data usually constitute one of the main content-bearing elements used for indexing Imago collections today. Descriptor data. Words or phrases concerning the subject matter in the" photograph. Natural language descriptions. Descriptions which capture the theme, general layout and arrangement of the photograph, objects pictured, and their relationships with other objects. This type of description has been the main concern of the present paper. Analysis of imagery. Measurements and delineation of aspects of the Image, such as identifying contours of bodies of water, determining distance between objects, etc. Conceptual maps. Indication of the content of the picture by means of linked structures. Such relational nets are used in present day question-answering systems having limited data base sizes. Line drawing extracts. Line drawings extracted from an image for expressing specialized relationships such as transportation networks, geologic boundaries, and water courses. These disparate elements must be tied together in such a way that queries are suitably responded to, with direct answers, extracts of imagery, measurements, or discussion provided to the user. A Dynamic Data Bank for Earth Resources Imagery Wo havo examined ^ the automated processing of descriptions to obtain encyclopedic entries for an image. Such automation can lead to what we have termed "dynamic data banks" for earth resources imagery. Experts using the imagery would return to the archival center their analysis and description of the imagery developed as part of their investigation. Such descriptions would be processed and merged, so that with time each image would have associated with it a combined or encyclopedic description. Of course, such descriptions would have to be tagged so that data on the background of the person making the description (his name, set, or field of specialty, analysis used, comments regarding the analysis, and reports produced using this image data) could be retrieved if desired. This "metadescriptive" material gives the context of the individual descriptions, the confidence that a user can have in the description, and links to related work. References 1. H. M. Gurk, C. R. Smith, and P. Wood, "Data Handling for Earth Resources Satellite Data," Proceedings of the 6th International Symposium on Remote Sensing of Environment. Center for Remote Sensing Information and Analysis, University of Michigan, Ann Arbor, Oct 1969
2

-

Ecological Surveys From Space, NASA SP-230,

B~2

LMSC-D246461 Part II
National Aeronautics and Space Administration, Washington, p. c . 1970 3. O. Firschein and M. A. Fischler, "Describing and Abstracting Pictorial Structures," Pattern Recognition Journal (in press, 197 1) 4. M. A. Fischler, "Machine Perception and DOHcription of Pictorial Data," Proceedings of the International Joint Conference on Artificial Intelligence, Washington, D.C., May 1969 5. O. Firschein, and M. A. Fischler, A Study In Descriptive Representation of Pictorial Data, 6-83-71-3, Lockheed Palo Alto Research Laboratory, Palo Alto, California, Feb 1971 (also, Second International Joint Conference on Artificial Intelligence, Sept 1971, London, England) 6. O. Firschein and M. A. Fischler, Descriptive Representations of Remotely Sensed Image Data, 6-83-71-5, Lockheed Palo Alto Research Laboratory, Palo Alto, California, May 1971 (complete version of the present short paper)

B-3