Mental Models for intelligent agents

1
Mental Models for intelligent agents

• Nikolaos Mavridis
• MIT Media Lab

2
Early motivation

• How are people able to think about things that
  are not directly accessible to their senses at
  the moment?
• What is required for a machine to able to talk
  about things that are out of sight, happened in
  the past, or view the world through somebody
  elses eyes (and mind)?
• What is the machinery required for the
  comprehension of a sentence like Give me the
  green beanbag that was on my left?

3
Overview

• Why mental models?
• Architecture
• W The descriptive language
• Reusable models
• Property description structures
• S Sensory structures
• F Instantiators / Predictors / reconcilliators
• Closing comments

4
Why mental models ?

• Goal Provide an intermediate representation,
  mediating between perception and language
• In essence
• an internalized representation of the state of
  the world as best known so far, in a form
  convenient for hooking up language
• a way of updating this representation given
  further relevant sensory data, and predicting
  future states in the absence of such data

5
Why mental models?

• But also
• A useful decomposition of a complex problem,
  suggesting a practical engineering methodology
  with reusable components, as well as a
  theoretical framework
• A unified platform for the instantiation of
  hypothetical scenarios (useful for planning,
  instantiation of situations communicated through
  language etc.)
• A starting point for experimental simulations of
• Multi-agent systems with differing partial world
  knowledge or model structure
• Primitive versions of theory of mind by
  incorporating the estimated world models of other
  agents
• Learning parameters or structures of the
  architectures, and experimenting with learned vs.
  innate (predesigned) tradeoffs (for example,
  learning predictive dynamics, senses-to-model
  maps, language-to-model maps etc.)

6
Notation FormalitiesD W, S, F

• D W, S, F A dynamical mental model
• W Mental Model State
• Wt state of the mental model at time t
• W the structure of the state (the chosen
  descriptive language for the world, ontology).
  Decompositions might be hierarchical.
• W O1, O2, into Objects/Relations
  (creations/deletions crucial)
• Oi P1, P2, into Properties (updates of
  contents but usually no creations/deletions)
• S Sensory Input
• St, S, S I1, I2, (Modalities/Sensors)
• F Update / Prediction function
• Wt1 F( Wt, St ) as a dynamical system
• F is a two-argument update / prediction function
• A decomposition (also Wht hypotheticals)
• (Wt,St)-gtWst (sensory-driven changes in W
  form)
• Wt-gtWpt (prediction-driven changes in W form)
• Wt1 R(Wst, Wpt) (the reconcilliation
  function)

7
Block Diagram (sync issues!)
8
W the descriptive language

• W in conversational setting include me, you,
  others
• Indexing Internal External Ids, continuity,
  signatures
• Bottom-up
• Simple_object (f.e. a cylinder)
• Object_relation (binary) (f.e. hinge joint)
• Compound_object SimpleObjectMap U
  ObjectRelationMap
• Agent Compound Object U Viewpoint U Gripper U
  Mover?
• Agent_relation (f.e. inter-agent joints,
  visibilty?)
• Compound_agent AgentMap U AgentRelationMap
  World
• Basic properties in simple_object,
  object_relation
• Property description structures (fixed, with
  confidences, with stochastic model, observational
  history, categorical form) - later

9
Simple_object

• class simple_object public Packable
• long ID
• //OBSERVER-INDEPENDENT PROPERTIES
• //Gradations of existence
• int exists //exists1 means sensory object,
  exists0 means virtual
• int body_exists //for ODE newtonian dynamics
• int geom_exists //for ODE collision-detection
• int draw_exists //should it be visible in the
  visualiser?
• //Position, rotation and velocity (second-order
  state space for rigid body)
• double pos3
• double R12 //remeber to set quaternion, too!
• double lvel3,avel3
• double facc3,tacc3 //force and torque
  accumulators...are they required?
• //Shape
• int shape define SOBJECT_SHAPE_BOX,
  SOBJECT_SHAPE_SPHERE 2, SOBJECT_SHAPE_CYLINDER 3,
  SOBJECT_SHAPE_CAPPEDCYLINDER 4
• double shapeparam3 //0 is also radius, 1
  is also length

10
Object_relation

• class binary_object_relation public Packable
• friend ostream operatorltlt(ostream os,
  binary_object_relation bor)
• public
• long ID
• long obj1ID
• long obj2ID
• //vectorltdoublegt params //e, tha ginei pio
  specific
• int type
• define BOR_TYPE_HINGE 1,define BOR_TYPE_HINGE2
  2,define BOR_TYPE_BALL 3,define BOR_TYPE_SLIDER 4
• double axis3
• define BOR_DIRECTION_X 1, BOR_DIRECTION_Y 2,
  BOR_DIRECTION_Z 3
• double anchor3
• double param10
• define BOR_PARAM_ANGLE 0
• define BOR_PARAM_ANGLERATE 1

11
Compound_object

• typedef mapltlong, simple_objectgt SimpleObjectMap
• typedef mapltlong, binary_object_relationgt
  BinaryObjectRelationMap
• static long compound_object_ID_counter0
  //Object 0 is not allowed!
• class compound_object
• friend ostream operatorltlt(ostream os,
  compound_object cobj)
• public
• long ID
• long signature //signature of IDs of component
  objects and relations
• int exists //existence flag....
• //should we allow existence of subobjects even
  if globally it doesnt exist?
• SimpleObjectMap objects
• long internal_object_ID_counter
• BinaryObjectRelationMap relations
• long internal_relation_ID_counter
• compound_object()

12
Agent, Compound Agent

• class agent public compound_object, public
  Packable
• public
• viewpoint viewpt
• gripper grip
• //mover mov
• public
• void pack(int initsend1)
• void unpack()
• typedef mapltlong, agent_ODEgt AgentODEMap
• typedef mapltlong, binary_agent_relation_ODEgt
  BinaryAgentRelationODEMap
• class compound_agent_ODE public Packable
• long ID
• long signature //signature of IDs of component
  objects and relations
• int exists //existence flag....
• AgentODEMap agents

13
More on the structures

• myObjects
• Packaging A dozen of .h/.cpp made into
  libmyobjects.a (some utils), include
  myobjects.h
• Expand/rethink types of relations!
• Think about joint (and relation!) recognition
  (Mann)
• myModels
• ready-made models for specific agents (ripley,
  human, environment) packaged in libmymodels
  Expand!!!
• These include parameter sets for customization,
  creation and deletion functions (as well as
  sensory update functions?). OuterIDs and body
  parts?
• myObjectsODE
• ODE-supplemented version for predictor

14
A model exampleripley_model.h

• EASILY PARAMETRISABLE FOR OTHER n-dof ARMS
• define SC 6.
• //Scaling factor, for some dimensions (rip etc.),
  not all!
• const double ripley_start_pos3
  23.2/SC,0,5.2/SC
• //Simple objects comprising ripley
• define NUM 6 //Number of links
  comprising ripley
• const double ripley_head_length 5/SC
• const double ripley_head_radius1 3.5/SC
• const double ripley_head_radius2 2.5/SC
• const double ripley_head_color3 .5,.5,.5
• const int ripley_head_texture DS_WOOD
• const double ripley_link_lengthNUM2.4/SC,
  22.8/SC, 9/SC, 1.8/SC, 2.3/SC, ripley_head_length
  /0.1/SC//5.2/SC/ // last one - length of
  camera
• const double ripley_color3 .61,.61,.61
• const int ripley_texture DS_WOOD
• const double ripley_link_radiusNUM2/SC, 2/SC,
  1.5/SC, 1.5/SC, 1.1/SC, ripley_head_radius2
• Etc

15
Property description structures

• The near future
• Class property_conf
• string name double value double confidence
• How to deal with ints/doubles and vectors?
• How to update conf? Decrease with time?
• 4-tier structure
• Class property_4
• categorical_descr c //variable granularity,
  context-sensitive boundaries
• property_conf ml
• stoch_descr distrib
• relevant_sensory_history senspointers
• Advantages
• Confidences vital for incomplete knowledge /
  information-driven sensing
• Homogenisation very useful for later experiments
  in feature selection etc.
• Suggestions/ideas?

16
S the sensory structures

• Vision
• Objectworld from 2D Objecter
• Extensions for 3D? Shape models? Partial view
  integration and the instantiator?
• Proprioception
• JointAnglesPacket form ripleys control
• Weight measurements
• Direct access to force feedback?
• Switching from continuous to on-demand feeding of
  new information (I.e. lookat() etc.)

17
F update/prediction functionI. Instantiators

• Modality-specific instantiators(updators/destruct
  .)
• Send create/update/delete packets to mental_model
• They SHOULD know previous world state
• R they modality or agent-specific?
• Should the generic agent models include specific
  sensory update functions?
• Virtual object instantiator
• Sometimes also used for creation of
  sensory-updated agents (I.e. self) boundaries?
• What would the clients need? Lets choose an API

18
F update/prediction functionII. Predictor
Reconcilliator

• Prediction rules
• Collision detection (collisions as obj_relat)
• Dynamics (reconcilliation with senses, inference
  of internal forces where to store?)
• Out of bounds deletions object stabilisation
• Reconcilliation
• How to resolve conflicts between sensed,
  predicted and requested? (think multiple sensors
  in car)
• Simplistic When no other info, use prediction.
  Else, blend senses with prediction?

19
Closing comments

• Many open questions / lots of work in the
  horizon!
• How do you achieve localisation of information
  and actions in these modules?
• Who should know what and how should things be
  synced? What about global signals sent from
  outside?
• Lets design for easy customisation/reusability.
  Significant parallelisation achieved.
• Some landmarks for the future
• Confidences in property descriptions
• Virtual object instantiator connected to 3D world
  creation tool for simulated external worlds
• Better shape description capabilities vision
• Connection to a different robot
• Two virtual agents in simulated world each with
  its own mental model and the estimate of the
  others simple demos
• 4-tier property descriptors
• Hypothetical scenarios and planning
• Parallel work
• Extend linguistic modules for more functionality
  given the richness of the structures
• Given confidences, better shape and extended
  bishop, do action (and speech) selection by
  maximum expected information return in general
  framework

20
Our ultimate goal

• Lets make ripley and his brothers more fun to
  talk to!
• And lets learn more about us on the way
• THANX 4 yr attn!