Spatial Language in Human-Robot Interaction

1
Spatial Language in Human-Robot Interaction

• Thora Tenbrink
• Project I1-OntoSpace

2
Structure

• Project Background
• Motivation
• General Research Questions
• Overviews
• Variation in object localisation (OL) tasks
• Variation in discoursal factors
• Range of variability in users utterances
• Analysis of projective terms
• Approach
• Default instructions and departures from that
  standard

3
Motivation

• Situated human robot interaction
• human instructs robot using natural language
• goals natural, successful, efficient interaction
• Difficulties
• different perceptual systems of human and robot
  necessity of mediation
• discourse situation influences linguistic choices
• determining factors as yet unknown

4
Reference systems (cf. Levinson 1996)

• Intrinsic
• Localisation of object by reference to the
  intrinsic properties of another entity, e.g.
  speakers front The ball is in front of me
• Relative
• Presence of relatum requiredThe ball is in
  front of the table
• Group-based reference if one of several similar
  objects is referred to The leftmost ball
• Absolute
• the earths cardinal directions (north, south)

5
Perspectives

• Speaker-centered
• Intrinsic The ball is in front of me
• Relative From my point of view, the ball is to
  the right of the table
• Absolute to the north of me
• Listener-centered
• Intrinsic The ball is in front of you
• Relative From your point of view, the ball is
  to the right of the table
• Absolute to the north of you
• Third-party point of view
• Intrinsic The ball is in front of Peter
• Relative Viewed from the churchs entrance,
  there is a bookshop on the right
• Absolute north of Bremen

6
Interpreting spatial language in context

• 9 kinds of underlying reference systems possible
  ? great potential for misunderstandings
• e.g. to the left misinterpreting the
  perspective yields the direct opposite
• Our approach Elicitation of data in experimental
  settings
• Analysis of linguistic choices in relation to
  situational parameters (spatial settings and
  tasks robot appearance and behavior)

7
Robots Pioneer Aibo
8
One scenario

Various configurations of similar and different
objects Various points of view No list of
commands Written and spoken mode
Pioneer robot
barrel

desk
test person
ca - mera
9
Experimental research questions

• Influence of scenario variables on natural
  language instructions
• What perspective is used?
• How are the locations of objects referred to?
• Group-based reference / intrinsic reference /
  distance / counting
• Landmarks / figures / rows
• Amount / sufficiency of information conveyed in
  instruction
• Users hypotheses about robot functionalities
• Effects of dialogue history
• Robot output
• Success vs. failure

10
Situational discoursal variability

• Variation in dialogic / interpersonal factors
• robot kind/ output/ functionality
• mode written/ spoken
• experimental instruction pointing/ reference to
  goal
• Variation in object localisation tasks
• number of objects
• positioning of objects
• robots view direction
• availability of unambiguous non-projective terms
  for reference

11
Reference systems

• Overlapping reference systems intrinsic regions
  roughly correspond to group-based regions
• Differing reference systems intrinsic regions do
  not overlap with group-based regions

12
Variation in object localisation tasks I

• LANDMARK without-landmark 24
  63.2
• with-landmark
  14 36.8
• --------------------------------------------------
  ----
• RELATION-GOAL- nearest-to-landmark 7
  18.4
• 
  closer-to-other-obj 3 7.9
• 
  equal-to-others 4 10.5
• --------------------------------------------------
  ----
• REFERENCE-SYST overlap-intrinsic-g 15
  39.5
• 
  diff-intrinsic-grou 23 60.5
• --------------------------------------------------
  ----
• PERSPECTIVE differ-robot-user 37
  97.4
• overlap-robot-user
  1 2.6
• --------------------------------------------------
  ----
• ROBOTS-VIEW-DI towards-objects
  17 44.7
• towards-user 11
  28.9
• both
  6 15.8
• none
  4 10.5

13
Variation in object localisation tasks II

• OBJECTS-PRESEN
• one 1 2.6
• two 10
  26.3
• three 20
  52.6
• four 7
  18.4
• --------------------------------------------------
  ----
• OBJECTS-INTRIN max-one-per-section 20
  52.6
• 
  more-leftright 8 21.1
• more-frontback
  10 26.3
• --------------------------------------------------
  ----
• PROTOTYPICAL-I
• no-prototypical 30 78.9
• left
  4 10.5
• right
  1 2.6
• front
  3 7.9

14
Variation in object localisation tasks III

• OBJECTS-BEHIND all-visible 35
  92.1
• one-behind
  3 7.9
• --------------------------------------------------
  ----
• GROUP one-clear-group 32
  84.2
• no-group
  3 7.9
• group-plus
  3 7.9
• --------------------------------------------------
  ----
• NON-PROJECTIVE not-obvious 25
  65.8
• available
  13 34.2
• --------------------------------------------------
  ----
• AVAILABLE-TYPE middle 9
  23.7
• class-name
  4 10.5
• --------------------------------------------------
  ----
• MIDDLE-OF same-class 6
  15.8
• superordinated-class 3
  7.9

15
Range of variability in users utterances

• Classification of levels
• not all utterances are instructions wie heißt
  du?
• not all instructions are goal-based vorwärts
• not all goal-based instructions are projective
  fahr zur zweiten Kiste
• Linguistic variability on different levels
• Syntax and politeness simple vs. complex
  imperatives du sollst zur Kiste fahren,
  presence vs. absence of politeness forms bitte
  (in instructions)
• contracted or full prepositions (in goal-based
  instructions) zur Kiste vs. zu der Kiste

16
Analysis of projective instructions

• Reference to the goal object via a projective
  spatial expression
• zur Kiste links, zur linken Kiste, zur Kiste
  vor dir
• Variability on three levels
• Modification of projective term
• with unmodified terms syntactic variation and
  variation between axes
• Complexity
• with single terms analysis wrt group-based,
  intrinsic, relative, or indeterminate reference
  systems
• Perspective

17
Assumptions

• No direct correspondence between reference
  systems and syntactic variation
• Kiste links could be either intrinsic or
  group-based
• linke Kiste sometimes assumed to be only
  group-based
• Which expressions are preferred (in which
  situation) for which kind of reference system?
• Applicability regions of intrinsic reference
  system larger than previously assumed overlap

18
Computational models Basis for coding of
reference systems
19
Method present status

• Qualitative analysis supported by statistical
  facts
• Looking for preferences and tendencies,
  dependencies and determining factors
• All projective China HRI instructions have been
  coded according to scheme
• extraction of relevant instructions via xml
• quick schematic coding with statistical results
  Systemic Coder
• Correlations between OL factors and instructions
• Looking for reasons for statistical results
  where there is systematic variation, there must
  be a reason
• situational (variability in OL tasks),
  discoursal, or individual
• Each variable in which OLs differ has been
  examined separately (only China files)

20
Results All files (1)

• MODIFICATION unmodified 346
  92.0
• modified
  30 8.0
• --------------------------------------------------
  ----
• PROJECTIVE-TER adjective 281
  74.7
• adverb
  57 15.2
• preposition
  8 2.1
• --------------------------------------------------
  ----
• AXIS left-right 315
  83.8
• front-back 31
  8.2
• --------------------------------------------------
  ----
• COMPLEXITY single-ref-sys 363
  96.5
• combined-terms
  13 3.5
• --------------------------------------------------
  ----

21
Results All files (2)

• REFERENCE-SYST group-based 58
  15.4
• intrinsic
  42 11.2
• indeterminate
  239 63.6
• relative
  24 6.4
• --------------------------------------------------
  ----
• RELATUM implicit 329
  87.5
• explicit
  34 9.0
• --------------------------------------------------
  ----
• PERSPECTIVE robot 361
  96.0
• user
  15 4.0
• --------------------------------------------------
  ----
• GIVEN-PERSPECT perspective-implicit 376
  100.0
• 
  perspective-explicit 0 0.0

22
Default projective instruction

• Fahre zur rechten Kiste
• Features adjective no modifications of
  projective term left-right axis indeterminate
  reference system implicit relatum robots
  perspective (implicit)
• Linguistic variation Kiste, Karton,
  Hindernis, Box, Kubus, Objekt etc. for
  reference abbreviated or full verb or verb
  omission (Fahre or fahr or nothing)
  contracted or full preposition (zur or zu
  der) politeness forms (bitte only
  exceptionally).
• Information structure unmarked Process
  (movement) is Given (unchanged throughout
  experiment), Participant (goal object) is New.
  Given part can be left out zur rechten Kiste
  (omission of verb)
• Departure from this default must have a reason
• fahre zu der Kiste, die rechts aussen steht vor
  dir

23
Some specific results Influence of OL variables

• Presence of landmark
• 17,6 relative reference systems where goal
  object is closer to landmark than to any other
  objects (even though configuration close to
  standard one) 24,2 adverbs
• 12,7 relative ref.sys. with equal distances
  12,7 adverbs
• no reference to landmark if other objects closer
  to goal object 2,9 adverbs
• Reference systems
• without overlap no indeterminate reference
  systems. 14,6 of projective terms are modified.
  9 prepositions, 12,4 adverbs, 64 adjectives
• with overlap 83,3 of utterances not clearly
  group-based or intrinsic. 5,9 of utterances are
  modified. No prepositions, 16 adverbs, 78
  adjectives

24
Robots view direction (OL4)

• Expectations where robot does not look at group
  centroid
• users prefer intrinsic reference system
• Result only slight difference 5,2 vs. 7,8
  intrinsic

• users assume a straight line as view direction
  between robot and group in order to employ a
  group-based reference system, yielding
  reference on the left-right axis
• Result (of all group-based or indeterminate
  instructions)
• box 3 93,9 left-right 6,1 front-back
• box 1 59 left-right, 35,9 front-back !

25
Explanations

• Front/back axis only possible if users shift
  the robots view direction in parallel towards
  the group centroid
• box 3 linker Karton is true for both intrinsic
  and group-based reference systems
• box 1 rechter Karton as well as hinterer
  Karton is only true for group-based reference
• it is not an intrinsic reference system the box
  is not behind the robot
• two active factors view direction and overlap of
  reference system

26
Non-projective terms in goal-based instructions

• Where possible in a given configuration, users
  prefer reference by simple use of class name to
  reference by projective terms.
• Fahre zur Kiste
• Where possible in a given configuration, users
  prefer reference to the middle object of the
  same class to reference via projective terms.
• Projective terms used in such a configuration are
  second choice and may differ from standard
  usage.
• Where a superordinate term is needed, users
  prefer projective terms but they also use the
  class name (incorrectly) Fahre zur mittleren
  Kiste

27
Problems

• Not enough data (analysed so far)
• often differences are not really due to the
  analysed factor, but can be traced back to
  something else
• some factors were not varied in the China
  experiment e.g., there was never more than one
  object in each intrinsic region
• Confounding of factors
• OL3 the only OL where there were only 2 objects
  (instead of 3), with only one object visible to
  the robot, and one object behind it, where there
  was no clear group of objects, and where the
  robot did not look at either the object(s) or the
  user
• Only situational factors examined but
  discoursal factors equally important
• Users preference of non-projective terms in some
  scenarios leads to untypical usages of projective
  terms when employed at all

28
OL3 Gehe zum linken Karton

• Situational factors
• there is no group of objects
• the object is not located in the prototypical
  left region of the robot
• robot is not oriented towards object(s)
• Discoursal factors
• previous success with adjectives
• not clear whether robot perceives object behind it

29
Default instructions

• Clear default scenarios
• Only one object class name
• 3 objects of same class with a clear middle one
  middle
• Groups with clear regions without competing
  objects simple projective adjectives
• When exactly do these defaults cease to be
  useful? Some expected but disproved limits
• no group of objects adjective
• non-prototypical intrinsic direction no
  modification
• no middle object of same class middle class
  name
• presence of landmark mainly no reference to
  landmark
• robot does not look at group centroid no
  preference of intrinsic regions
• But the more the scenario departs from the
  default configuration, the less the default
  instructions are used.

30
Interpreting spatial instructions

• Generous interpretation necessary
• Higher-ranking condition Sufficient contrast to
  competing objects in order to identify goal
  object
• For humans it is usually clear what is meant
• in natural human-human interaction as well as wrt
  human overhearers in our robot experiments
• We need to specify these human intuitions
  sufficiently for a robot to reach the same
  conclusions
• Understanding production processes and strategies
  supports interpretation
• More complex scenarios will show more of users
  strategies where default instructions fail