B. Chandrasekaran, Bonny Banerjee, Unmesh Kurup, John Josephson

1
Diagrammatic Reasoning in Army Situation
Understanding and Planning Architecture for
Decision Support and Cognitive Modeling

• B. Chandrasekaran, Bonny Banerjee, Unmesh Kurup,
  John Josephson
• The Ohio State University
• Robert Winkler
• US ARL

2
Outline of the Talk

• What is Diagrammatic Reasoning? Why is it
  important in for Army Decision-Making?
• Basic research issues brief outline of
  progress
• Representation, Architecture
• Technology built on Science Applications built
  on technology
• Some Remarks on the Future

3
Ubiquity of Diagrams in Army Operations

• The Army is about space
• taking it, defending it, controlling it, avoiding
  it, going through it
• Army planning, situation assessment, situation
  monitoring, fusion, all use diagrammatic
  representations
• Standards for symbols to be used are defined in
  FMs

4
How DR Research Can Help Provide More Effective
Decision Support

• Automating repetitive routine reasoning tasks
  that involve diagrams
• E.g. Critiquing COAs for vulnerability to ambush
• Better interface design. Understanding what makes
  a diagram good, i.e., makes relevant information
  readily available, without error, can help in
  design of decision interfaces
• Requires how human cognitive architecture
  perception work in performing diagrammatic
  reasoning
• Cognitive modeling to evaluate diagrammatic
  interfaces

5
What Diagrammatic Reasoning is...

• DR is reasoning, i.e. making inferences and
  problem solving with visual representations, and
  involves a collaboration between two systems
• A symbolic reasoning system that combines
  information from the diagram with other
  information to make inferences, to set up
  diagrammatic perception and action subgoals
• A diagrammatic representation from which
  perception obtains information about spatial
  relations and properties re. diagrammatic objects
• .

The phenomena of interest can also take place in
our imagination
6
Diagrammatic Reasoning is Not..

• What it is not
• It is not image processing (such as processing
  satellite images for objects of interest, though
  DR can be used as part of it)
• Image processing may be required to extract the
  diagram from an image
• It is not computes graphics, though diagrams can
  often be usefully superimposed on such pictures
• It is not parallel array processing algorithms
  that solve problems such as shortest paths,
  though there is a role for such algorithms in the
  overall process of diagrammatic reasoning,

7
Some Scientific Issues weHave Made Progress In

• What is a diagram as a representation
• Specificity of a diagram. How are we able to
  solve a general problem from a specific diagram?
• Representation in the mind in the computer.
• The nature of the architecture that can perform
  diagrammatic reasoning
• Opportunistic integration of diagrammatic
  inferential operations
• How do diagrams get into long-term memory?
• How are diagrams composed to make new diagrams?
• Abstraction of diagrams

8
Computational Model of Diagrammatic Reasoning

• Recall two reasons we mentioned, to develop
  computational frameworks for diagrammatic
  reasoning
• Automation or semi-automation.
• Building cognitive models
• Good News!
• A computational architecture that can be used for
  automation can also be used for modeling.
• Our bimodal cognitive architecture BiSoar

9
Symbolic Inference Perception from Diagrams

• Any system that can support symbolic
  representation inference can be integrates with
  our DRS.

• Soar Act-R happen to be symbolic reasoning
  systems with especially useful properties for
  general intelligence.

10
BiSoar a Bimodal Cognitive Architecture

• Thinking has been usually modeled in AI
  Cognitive Science as syntactic operations on
  abstract symbols. Soar, Act-R, etc.

• BiSoar keeps the general architecture, but all
  states can be bi-modal The agent can have both
  linguistic pictorial representations

11
Diagrammatic Representation System

• Diagrams consist of three types of objects
  Points, Curves Regions.

• Diagrams are not just images, they are a spatial
  configuration of spatial objects.

12
The Role of Perception

• Perception and Action Routines A set of
  algorithms that create or modify diagrams and
  perceive objects and spatial relations between
  elements in the diagram.

Perception/Action Routines
Perception/Action Routines
Diagram Representation in DRS
13
Perceptual Routines Recognize Emergent Objects
and Relations
Base set domain-independent, open-ended

• New object recognition and extraction routines
• Intersection-points between line objects, region
  when a line closes on itself, new regions when
  regions intersect, new regions when a line
  intersects with a region, extracting
  distinguished points on a line (such as end
  points) or in a region, extracting distinguished
  segments of a line (such as those created when
  two lines intersect), extracting periphery of a
  region as a closed line. Reverse operations are
  included such as when a line is removed,
  certain region objects will no longer exist and
  need to be removed.
• Relational perception routines Inside (I1,I2),
  Outside, Left-of, Right-of, Top-of, Below,
  Segment-of (Line1, Line2), Subregion-of (Region1,
  Region2), On (Point, Line), and Closed (Line1).
• Translation, rotation and scanning routines may
  be combined with routines in 1 and 2. Example,
  Intersect (Line, Rotate (90deg, Line 2)).

14
Action Routines

• Create diagrammatic objects, such as a path that
  goes from point1 to point2 while avoiding
  region2. Path finding and path modification
  routines are especially useful in Army
  applications.

15
Automatic Synthesis of PRs ARs

• Banerjees Ph. D Thesis gives many techniques for
  automatic synthesis of PRs ARs. Example In
  the situation below, where c is a wall, A is a
  member of Red force, where can BT a member of
  Blue force hide?

• Once the problem is converted to the language of
  geometry, the set of all points p such that line
  Ap intersects c, his techniques can automatically
  construct algorithms to solve the problem.

16
Attention, Learning, Memory

• BiSoar can be parametrized to mimic the
  limitations of human attention short term
  memory.
• BiSoar can learn by a mechanism called
  chunking. As a result of attention short
  term memory limitations, BiSoars LTM contains
  smoothed approximations of complex shapes.

17
Example of Automating DR

• Entity ReIdentification in ASAS All Source
  Analysis System
• Currently very human-analyst labor intensive, and
  many sightings are simply left unattended

18
Diagrammatic Reasoning in Information Fusion in
an ASAS Problem

• The task is to decide for a newly sighted entity,
  T3, which of the previous sighted identified
  entities it is.

Regions impassable for vehicle types of interest
are marked and represented diagrammatically in
the computer
19
Entities from Past Sightings Retrieved
Two tanks, T1 and T2 were retrieved along with
their locations and times of sighting
The Fusion Engine asks for ways in which T1 T2
could have gotten to the location of T3 within
the available time
20
Architecture Combines Symbolic Diagrammatic
Reasoning
For T1 T2, DR finds eight possible routes, but
rules out all but one. The figures shows the
routes for T1 T2.
21
Example of Action Routine
The Database reveals that there are sensor fields
but they didnt report any vehicle crossings.
A similar question about T2 reveals that T2 also
crossed a sensor field, which also didnt report
any vehicles. However, DR says T2 could not have
avoided the sensor field.
22
Numerous Other Applications

• Rerouting
• Ambush vulnerability analysis
• Plan critiquing in general
• Other uses in information fusion, where the
  hypothesis has significant spatial components

23
Examples of Cognitive Modeling

• Kurups thesis models explores
• How errors in geographical recall come about.

• Recalled spatial relationships between
  geographical entities show distortions
• Ex What is the relationship between San-Diego
  and Reno?

24
Three Models

• Model 1 Agent has complete map
• Model 2 Agent has symbolic knowledge that SD is
  South of SF and Reno is East of SF.
• Model 3 Agent has knowledge that SD in
  California, Reno in Nevada and that California is
  West of Nevada.

25
Models or Route Recall Graph Comprehension

• Loss of detail in recall of routes
• Kurups Model posits attention limits as
  explanation
• Graph Comprehension
• Leles BiSoar model unifies a variety of observed
  phenomena
• Using external graphs requires mental imagistic
  operations!

26
DR Automation Modeling Central to Decision
Support

• The research reported here has laid come
  scientific technological foundations of this
  area.
• Has also built some demonstration applications
  models.
• But its still a baby, theres potential, but
  needs to be nurtured to produce full benefit.
• Many important research issues
• Extraction of DRS from physical diagrams
• How are appropriate diagrams to help solve
  problems generated?