Reactive and Responsive Intelligent Environments

1
Reactive and Responsive Intelligent Environments

• Kevin Quigley
• aire group
• MIT AI Lab

2
Reactive and Responsive Environments

• We are trying to build pervasive, perceptually
  enabled human-centered environments
• Such an environment must respond in reasonable
  ways to high level requests from its users.
• It should be up to the system to figure out a
  reasonable way to implement the request,
  translating goals to plans that meet the users
  needs and that utilize available resources.
• E.g. I might ask to light the room up the system
  responds by opening the drapes.
• Such an environment should react to events in the
  environment even when there is no explicit user
  request.
• E.g. when I walk into my room in the morning, the
  system should light up the room.
• Reactions and responses should both be
  contextually sensitive.
• Both must show human levels of adaptivity

3
5 Challenges

1. Providing a practical level of knowledge
   representation that enables groups interactions
   and grounding in the real world of space and time
2. Providing run-time composable services in a
   multi-user environment that make optimal use of
   the currently available resources
3. Recovering from equipment failures, information
   attacks, misestimates of sensors, etc.
4. Coordinating and fusing information from many
   sensors and modalities
5. Capitalizing on and recognizing context (task,
   location, personal style state)

• 6. Maintaining security and privacy and trading
  these off against other goals

4
Challenge 1 Grounding in Real-World Semantics

• We want to build applications that service many
  individuals and groups of individuals
• These people will move among many physical spaces
• The devices and resources they use change as time
  progresses
• The context shifts during interactions
• The relevant information base evolves as well.
• The system is required to respond dynamically

5
Research Agenda Knowledge Representations

• People
• Interests, skills, responsibilities,
  organizational role
• Organizations
• Members, structure
• Spaces
• Location
• Subspaces
• Devices and resources
• Resources

• Information nodes
• Topic area, place in ontology, format
• Services
• Methods, parameter bindings, resource
  requirements
• Agents
• Capabilities, society, acting on behalf of whom
• Events
• E.g. Person identification, motion into a new
  region of space, gestures
• Qualitative Changes in any of the properties in
  the KR

6
Challenge 2 Adaptive Resource Management

• In most systems, applications are written in
  terms of specific resources
• (e.g. The left projector in Michaels office, or
  worse yet, a physical address).
• This is in conflict with
• Portability across physical contexts
• Changes in equipment availability across time
• Multiple applications demanding similar resources
• Need to take advantage of new resources
• Need to integrate mobile devices as they migrate
  into a space
• Need to link two or more spaces
• What is required is a more abstract approach to
  resources in which no application needs to be
  tied to a specific device.

7
An Example of Service Requests

• When I come into my office in the morning its
  dark
• The lights are out
• The drapes are closed
• I ask the office to light up the room
• Its a sunny day, it opens the drapes
• If I had asked it to turn on the lights, it
  wouldnt have opened the drapes
• Its a cloudy day, it turns on the lights

8
Responsive, Goal-Directed Processing
Goals
Service mapping is provided by the resource
management component
9
Services are Dynamically Mapped to Plans
The System Selects the Plan Which Maximizes Net
Benefit
10
Challenge 3 Robustness and Recovery From
Failures

• Breakdowns are inevitable
• Resources sometimes fail while being used
• The system acts on sensor data which has
  uncertainty
• The system renders services by translating them
  into plans
• A plan-monitor watches over the execution of a
  plan.
• Each plan step accomplishes sub-goals needed by
  succeeding steps
• Each sub-goal has some way of monitoring whether
  it has been accomplished
• These monitoring steps are also inserted into the
  plan
• If a sub-goal fails to be accomplished,
  model-based diagnosis isolates and characterizes
  the failure
• A recovery is chosen based on the diagnosis
• It might be as simple as try it again, we had a
  network glitch
• It might be try it again, but with a different
  selection of resources
• It might be as complex as clean up and try a
  different plan

11
The Plan Monitor Manages Recovery From Failures
I dont see light on the screen

• Plan 1
• Locate a systems wizard in the E21
• Monitor check that person is still there
• Turn on the selected projector
• Monitor check that projector turned on
• Project the message
• Done
• Monitor check that the person noticed the message

Plan Breakdown
Projector-1 must be broken. Well try again, but
using Projector-3.
I see Sally by the screen
12
Making the System Responsible for Achieving Its
Goals
Diagnostic Service
Service Request
Plan-for
Localization Characterization
Repair Plan Selector
alerts
Scope of Recovery Selection of Alternative
Rollback Designer
Concrete Repair Plan
Plan Monitor
Resource Allocator
Resource Plan
A plan is a partially ordered collection of
steps Each step achieves a subgoal Some steps
establish pre-requisites for others
Enactment
13
Self-AdaptiveGoal Directed Processing
14
Challenge 4 Context Awareness

• The context should influence how the system
  behaves
• Task Structure
• Location
• Emotional State
• Personal Style
• Perception can help determine the context
• The system should choose its reactions to events
  based on the context
• Perceptual interpretation should be biased by
  context
• E.g. a person near the White Board, is likely to
  start drawing
• Estimation of utility should be influenced by
  context

15
A Reactive System Responds to Events
If somebody enters the room Then illuminate the
room
16
Structure of REBA

• A Reaction maps an event to a goal
• Reactions are grouped into behavioral bundles
• Sets of reactions that are always activated as a
  unit
• The Context is determined by the task within a
  plan
• As well as location, people present, ...
• Each context activates a set of behavioral
  bundles
• Contexts have Sub-Contexts, activities that occur
  within other activities
• Watching a video within a meeting
• The active reactions of a sub-context override
  the reactions of the parent context

Goal2
Sub-context-1
Goal1
Context Stack
Sub-context-2
Reactions
17
Context Awareness Conditions Reactions
If somebody enters the room Then illuminate the
room
But not if a movie is being watched
18
Context Sensitivity
Reactive Processing is provide by the REBA
MetaGlue Component
19
Challenge 5 Perceptual Integration

• We want to separate the implementation of
  perceptual tasks from the uses to which
  perception is put
• Some modules advertise the class of behavioral
  events they are capable of recognizing and
  signaling
• These events are organized into a taxonomy
• The same event can be signaled by quite different
  perceptual modules (e.g. both face and voice
  recognition can localize a person).
• Other modules register their interest in certain
  classes of events
• Requests at a higher level in the taxonomy
  subsume lower level events
• Modules which receive low-level events may
  register for and collate many different classes
  of events
• They combine these and signal higher-level events
• Modules may request perceptual services when they
  are uncertain of their conclusions

20
A Dynamic Event Bus For Perceptual Integration
Interested in the Location of Individuals Signal
People Approaching the Whiteboard
White Board Context Manager
Interested in Body Motion Signal the Location of
Individuals
Interested in Body Motion I Signal Face Location
Voice Identification
Face Spotter
Signal Body Motion
Visual Tracker

• A Blackboard System
• Publishers Subscribers Are Knowledge Sources
• Events Are the Blackboard Data Items
• Highly Distributed
• Use of Bayesian Techniques

21
Blackboards Using Context
Context Discussion of Oil Tanker Crash Fragility
of Environment
Its not Hard to Wreck a Nice Beach
Recognize
Speech
Wreck A Nice Beach
Wreck A NICE
Recognize
A NIS
Wreck
Rec
Og
IZE
R eck
Og N
I Z
SP
22
Integration of Reactive and Goal Directed
Processing
Goals
Reactions
Diagnosis Recovery
Service Mapper
Plan Monitor
Events
Context
Plans
Resource Pool
Resource Allocator
Blackboard
Sensory Systems
Actions
23
Summary

• Responsive processing dynamically maps goals to
  plans
• Plans are selected by balancing the benefit to
  the user against the cost of resources
• Plan monitoring recovers from plan breakdown
• Access control is handled as part of the cost
  benefit analysis
• Reactive processing dynamically maps event to
  goals
• Events are handled within context
• Perceptions are maps to events within context

24
Reactive and Goal Directed Processing
with Security
Goals
Reactions
Diagnosis Recovery
Service Mapper
Plan Monitor
Events
Context
Plans
Resource Pool
Resource Allocator
Blackboard
Access Policies
Sensory Systems
Actions
25
The Metaglue System

• Software agents for intelligent spaces

26
Computational needs

• Smart Environments
• Distributed Components
• Dynamic Changes
• Highly Varied Equipment
• New Modes of Interaction
• Frequent Failures
• Abundance of Information

• Metaglue
• Communication
• Resource Management
• Customization
• Multi-modal HCI
• Agent Recovery
• Persistent Storage

27
The Metaglue System

• What is it?
• A communication infrastructure for building
  systems of distributed software agents
• A software architecture for creating adaptive
  applications for Intelligent Environments
• Why for Intelligent Environments?
• dynamic adaptation of the applications based on
• the availability of resources (hardware/software)
  in the current system/environment
• security controls of the participants
• preferences of the participants

28
Metaglue Software Infrastructure

• What do you get with Metaglue?
• Multi-modal Human-Computer Interaction
• Spoken Natural Language
• Perceptual (Vision)
• Direct manipulation (Graphical)
• Persistent storage
• Multiple communication paths
• Remote procedure call (Java RMI)
• Publish/Subscribe message passing
• Configurable setting for asserting preferences
• Start on demand of Agents
• Automatic recovery / handling of direct
  communication errors
• Resource Management and Service mapping

29
What Are Software Agents?

• Our definition
• A software agent is any software object capable
  of communication by exposing functionality to
  other agents running within the network.
• Metaglue is a Multi-Agent System where agents
  perform individually specialized, (usually)
  simple tasks but connect in a web of
  intercommunication to cooperate on more complex
  tasks.

30
More than Java Objects

• Metaglue Agents can be as complex or simple as
  necessary
• Larger programs can be used in Java thru JNI or
  other interface
• ViaVoice, ASSIST (a sketching tool)
• Most Metaglue Agents are simpler and
  single-purpose
• Projector display or light control
• Complex agents are those that control other
  simpler agents
• ReBa Reactive Behavioral System
• SPIe Self-adaptive Plan-based Intelligent
  Environment
• Metaglue provides to agents
• identity and occupation
• the entity this agent represents
• location and the ability to change locations
• intrinsic communication to other agents

31
What are some of the agents in Metaglue?

• Device Control
• lights, drapes, fans, sensors
• DVD CD players, MP3
• video multiplexers, projectors
• Cameras, TVs, VCRs, Audio
• Data Organization Systems
• Blackboard agents data flow
• START system NLP KB
• Presence Location agents
• Newswall visual data organization and
  presentation

• Agent Systems Applications
• ReBa interactive behaviors
• SPIe and Planlet plan monitoring
• Web info display
• Debugging and Logging
• Agent testing, Simulated devices
• Log Catalog Monitors
• Notification listeners
• Recreation
• Checkers, Boggle, RPG, HexaCheckers, Crosswords,
• ELIZA clone

32
System Organization - Societies

• Clusters of agents that operate on behalf of a
  real-world entity (space or person) are called
  Societies
• Societies allow the same agent to exist with
  different customized attributes.
• Agents can talk to agents in other societies as
  easily as their own society
• Societies look like agents when viewed from the
  outside
• They exposed functionality to higher level
  resource management through Hyperglue

33
System Organization - Catalog

• The Catalog Agent is the central component which
  knows about all running agents

34
System Organization - MetaglueVM
To Catalog

• The Metaglue Virtual Machine (MVM) runs as a base
  platform for all other agents in the Metaglue
  system.
• It handles all registration of the agent with the
  Catalog Agent
• Provides methods for direct communication (RMI)
  to other agents on the current catalog

35
Calling on Other Agents agent present

1. A reliesOn call by Agent B will create a chain of
   events to get the RMI stub for Agent A
2. The MVM takes the call and passes it to the
   Catalog Agent
3. The Catalog checks the internal table of agents
   to see if there is one matching the requested
   description
4. If the agent exists, the RMI stub for Agent A is
   returned

36
Calling on Other Agents agent not present

• 3. The Catalog Agent checks the internal table
  of agents to see if there is one matching the
  requested description
• If the agent does not exist, it must be started
  locally (calling MVM)
• The new Agent A registers its stub
• That stub is then returned to Agent B

37
Automatic Agent Recovery

• Failure recovery through proxies

38
Without error handling
39
Without error handling
40
Without error handling
41
Without error handling
42
Error handling with proxy objects
43
Error handling with proxy objects
44
Error handling with proxy objects
45
Statistics on Metaglue

• 10 Tons of fun
• There are over 450 agents that exist within
  Metaglue
• Between 50 and 80 agents are running the
  intelligent room
• You are using more than 10 agents just while
  running the X10BasicLightControl
• Test it! Use agentland.util.LogMonitor
• Metaglue has been in development since 1998
• The system is used in several offices and homes
  including the office of the AI lab director,
  Rodney Brooks
• There are 2 full spaces at MIT (a 3rd is coming
  soon!) and one space in Australia running
  Metaglue
• Why not get your own?