Quals Context Fabric

1
Context Fabric Infrastructure Support for
Context-Aware Systems
Jason I. Hong Qualifying Exam Proposal
2
Why Context-Awareness?
Context Types
Existing Examples
Human Concern
Room Activity
Smoke Alarm
Safety
Room Activity
Auto Lights On / Off
Convenience
Object Identity
Barcode Scanners
Efficiency
Personal Identity Time
File Systems
Finding Info
Time
Calendar Reminders
Memory
3
Technology Trends

• Sensors
• GPS, Active Badges, Active Bats
• Smart Dust
• Cameras and microphones
• Recognition algorithms
• MSR Radar location from 802.11
• Smart Floor footstep force (Orr Abowd 2000)
• Wireless technologies
• Bluetooth, 802.11, cell phone

4
Why Context-Awareness?
Existing Examples
Context Types
Potential Examples
Human Concern
Activity
Safety
Identity Time Location Proximity
Activity History
Smoke Alarm
Health Alert
Activity
Convenience
Auto Lights On / Off
Auto Cell Phone Off In Meetings
Identity
Efficiency
Barcode Scanners
Service Fleet Dispatching
Identity Time
Finding Info
File Systems
Tag Photos
Time
Memory
Calendar Reminders
Proximal Reminders
5
A New Class of Context-Aware Apps
Cyberguide (Abowd et al)
Active Badge (Olivetti)
ParcTabs (Xerox PARC)
6
Problems with State of the Art

• Coordinating all the context
• Will have many distributed, heterogeneous
  sensors, services, and devices out there
• Won't be designed a priori for interoperability
• Building robust context-aware apps
• Dynamically changing computing environment
• Sensors, services, and devices will vary in
  capability and availability
• Developers are forced to devote a lot of time and
  energy to non-application issues

7
Proposed Meta-Solution

• Shift burden of context-awareness from individual
  devices onto infrastructure
• Discovering, Acquiring, Transforming, Refining,
  Routing, and Reacting to Context
• Provide services that can be easily deployed
  everywhere and used by any device
• Have simple, uniform system interfaces
• Any device with a network connection can discover
  and use these services
• Abstract out messy details of sensors and services

8
Thesis Statement

• A set of core infrastructure services coupled
  with a specification language for stating context
  needs will reduce the effort in developing robust
  context-aware applications.

9
Proposed Solution
App
Agents
Application Layer
Device Agents
Context Specification Language
User Agents
Services
Context Event Service
Context Query Service
Context Layer
Automatic Path Creation
Logical Sensor Layer
Sensor Management Service
Physical Sensor Layer
10
Research Contributions

• A way of explicitly specifying context needs
  independently of specific sensors and services (
  ontology for doing it)
• Automatically assembling a dataflow from
  heterogeneous sensors and services at run time to
  satisfy context needs
• Seeing how far queries, events, and data flow
  processing can go for context

11
High-Level Plan (Mir Space Station View)

• Design Context Specification Language
• Design, implement, and deploy core context
  services
• Evaluate by implementing several simple-to-medium
  complexity apps on top
• Re-iterate design and implementation based on
  what was learned

12
Outline

• Background and Motivation
• Problem and Proposed Solution
• Context Fabric Architecture
• Evaluation
• Related Work
• Proposed Schedule
• Summary

13
Context Specification Language

• ltcontext version1.0 typesubscribegt
• ltrequestorgt
• requestor identification
• requestor authentication
• requestor input
• lt/requestorgt
• ltrequestgt
• context need
• lt/requestgt
• lt/contextgt

14
Context Specification Language

• Need to express context about and relationships
  between People, Places, Things
• Predicates
• Identity (Who is? What is? Is with?)
• Location (Near? Nearest? Distance? Path?)
• Activity (Is busy? Is in meeting? Current task?)
• Time (In past? In present? In future? On date?)
• Some of this vocabulary done by Schilit
• Implicitly encoded in his APIs
• One goal is to extend his work in spec language
• Another is to make it extensible for future
  context types

15
Context Specification Language

• Common parameters
• Max number of results wanted
• Return name
• Return data type (e.g. String, List, Table)
• Minimal probability of correctness desired
• Relevant sensor input requestor has
• Event parameters
• Event rate (e.g. at most 1 event per second)
• Event callback (e.g. RPC, socket port)
• Max number of events desired
• Granularity of change (e.g. 1 meter)

16
Context Event ServiceContext Query Service

• Uniform abstraction for apps
• Pass a Context Specification to Context Event or
  Context Query service
• Everything is handled by infrastructure
• Event and queries two sides of same coin
• Query is synchronous and one-time
• Event is asynchronous and repeating
• Bulk of work is done by Automatic Path Creation
  though

17
Automatic Path Creation

• A way of assembling small services into larger,
  more interesting ones
• Operators
• Paths

18
Automatic Path Creation

• Automatically assemble path based on resources
• Don't have to know specific sensors and services
• Easier to add new ad hoc functionality
• Will need a Service Description Language

19
Agents

• Re-implement from ParcTabs (Schilit 1995)
• Device Agents
• Bridge for physical devices into infrastructure
• Holds context state about devices
• Centralizes context to and from device (better
  for preferences and filtering)
• Job is to handle all context issues that
  cannot / should not be handled on device itself
• User Agents
• Holds context state about people
• Preferences, location info, filters, etc

20
Context Fabric ExampleContext-Aware Tour Guide

• Mobile PDA tour guide
• Display map of local area
• Display nearby points of interest
• Bring up info on proximity
• Display location of others
• Capture history

21
Context Fabric ExampleContext-Aware Tour Guide
Palm Pilot
Palm Pilot
Cell Phone
Infrastructure
22
Context Fabric ExampleContext-Aware Tour Guide
1. Display map of local area
1. Turn on device, local GPS registers with
Sensor Management Service
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
23
Context Fabric ExampleContext-Aware Tour Guide
1. Display map of local area
2. Start up tour guide application
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
24
Context Fabric ExampleContext-Aware Tour Guide
1. Display map of local area
3. Setup context query for changes in location
(granularity depends on if indoors or outdoors)
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
25
Context Fabric ExampleContext-Aware Tour Guide
1. Display map of local area
4. Receive location updates
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
26
Context Fabric ExampleContext-Aware Tour Guide
1. Display map of local area
5. Display location updates
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
27
Context Fabric ExampleContext-Aware Tour Guide
2. Display nearby points of interest
1. Periodically query for nearby points of
interest and Cache results (e.g. restaurants,
landmarks, etc)
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
28
Context Fabric ExampleContext-Aware Tour Guide
2. Display nearby points of interest
2. Local Context Query Service delegates to
Automatic Path Creation Service
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
29
Context Fabric ExampleContext-Aware Tour Guide
2. Display nearby points of interest
3. Automatic Path Creation Service takes GPS
info and tries to answer it locally (fails).
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
30
Context Fabric ExampleContext-Aware Tour Guide
2. Display nearby points of interest
4. Local APC delegates to infrastructure APC
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
31
Context Fabric ExampleContext-Aware Tour Guide
2. Display nearby points of interest
5. Infrastructure APC looks for services that can
provide location-based points of interest info
or parts of this info
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
32
Context Fabric ExampleContext-Aware Tour Guide
2. Display nearby points of interest
6. Infrastructure APC returns relevant
information (or "failure due to x" if it cannot)
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
33
Context Fabric ExampleContext-Aware Tour Guide
3. Bring up info on proximity
1. Setup proximity events on points of interest.
Attach notes to event that display information
about point of interest.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
34
Context Fabric ExampleContext-Aware Tour Guide
4. Display location of others
1. Subscribe to location change events from
others.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
35
Context Fabric ExampleContext-Aware Tour Guide
4. Display location of others
2. Can't process locally, subscription request
forwarded to mirroring Device Agent.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
36
Context Fabric ExampleContext-Aware Tour Guide
4. Display location of others
3. Device Agent makes individual subscription
requests.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
37
Context Fabric ExampleContext-Aware Tour Guide
4. Display location of others
4. Individual device agents make a subscription
request to their mirroring devices.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
38
Context Fabric ExampleContext-Aware Tour Guide
4. Display location of others
5. Location change events are routed back to
tour guide and displayed.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
39
Context Fabric ExampleContext-Aware Tour Guide
5. Capture history
1. Just store all interesting events locally.
Palm Pilot
E
Device Agent
Q
Palm Pilot
Q
E
Map Database
E
E
Tour Guide App
Device Agent
Device Agent
Cell Phone
E
Q
Q
Q
Context Event Service
Context Query Service
Automatic Path Creation
Automatic Path Creation
Sensor Mgt Service
Service Discovery
GPS
40
Outline

• Background and Motivation
• Problem and Proposed Solution
• Context Fabric Architecture
• Evaluation
• Related Work
• Proposed Schedule
• Summary

41
Evaluation

• Iterative design process
• Work out designs, deploy, build apps, refine
• Initial evaluation apps will be existing ones
• Reimplement some existing context-aware apps
• Informally evaluate apps for robustness
• Informally evaluate apps for defaults (e.g.
  privacy policy)
• Continued evaluation will be novel apps
• Focus on robustness and defaults again
• Also get others to build context-aware apps

42
Related Work

• ParcTab System (Schilit 1995)
• Device Agents, User Agents, Active Maps
• Plan to re-implement these and new ones too
• Context Toolkit (Dey 2000)
• Widgets, Interpreters, Aggregators
• Much is subsumed in Automatic Path Creation
• Interactive Workspaces (Fox et al 2000)
• Paths path creation and blackboard architecture
• Self-describing and extensible events

43
How Context Fabric Differs

• Separate what context is needed from how it is
  processed via Context Spec Language
• Shift processing from static to dynamic
• Say what you need at design-time
• Figure out how to get it at run-time through
  Automatic Path Creation

44
Proposed Schedule

• Phase 1 (0-8 months / PARC)
• Initial design and implementation of Context
  Specification Language and core services
• Evaluate with simple context-aware apps
• Phase 2 (8-12 months)
• Re-iterate and incorporate what was learned
• Release source code
• Evaluate with medium complexity apps
• Phase 3 (12-18 months)
• Incorporate more feedback (smaller changes)
• Write dissertation

45
Exit Conditions and Deliverables

• Spec for Context Specification Language
• Basic ontologies for context
• Expressive enough for majority of context apps
• Working core services
• Context Query, Context Event, Automatic Path
  Creation, plus basic operators
• Service Description Language for APC operators
• Source code available for general use

46
Summary

• Infrastructure approach to context-awareness
• Context Specification Language for stating
  context needs
• Context Event Service, Context Query Service as
  primary APIs
• Automatic Path Creation for processing
• Evaluation focused on defaults and robustness

47
A New Cal Record!!

• Only one day to schedule quals
• through email
• with a professor at Stanford even!

48
Extra Slides
49
What is Context?

• Situated Action (Suchman)
• Responsiveness, improvisation to situation
• Actions are moment-by-moment, highly fluid on
  situation
• Activity Theory (Nardi)
• Subject, Objective, Operations, Artifacts, Env
• Activity defines context but it changes as list
  above changes
• Context depends on sensing and inferring, but
  difficult to know people or goals

50
What is Context?

• Provides fair criticisms on existing
  context-aware apps
• Response Focus on the more routine and
  predictable actions
• Response Focus on simple first-order approx
• Examples Smoke alarms, Calendar reminders
• Response Is occasionally wrong, but won't get
  better unless we do more research here
• Response Infrastructure enables more research in
  this area

51
What is Context?

• Merging of virtual with physical and social
• Mostly implicit input, often secondary to
  task at hand
• People / Places / Things / Virtual / Time
• People Location, Identity, Task, Affect
• Places Activity, People, Temperature, Audio
• Things Location, Identity
• Virtual Services, Bandwidth
• Time All of these Past, Present, and Future

52
What is Context?

• But it's a rathole to try to define it
• Workflows? (Greenberg 2001)
• Intent and Desire?
• Natural Language Processing?
• What isn't context?
• Instead, operationalize it
• Distributed, multi-device, sensor-based
• Fairly well-defined and computable concepts (e.g.
  location and identity, but not intent or workflow)

53
Why Context-Awareness?

• Sensors are coming
• Increase channels of input / output

Graphics Sound
Mouse Keyboard
54
Why Context-Awareness?
Skeptics toward the ubiquitous computing movement
quickly point out that those involved in
promoting Mark Weiser's initial vision are mainly
interested in pushing technology As a reaction
against this fair criticism, many researchers
spend countless hours trying to dream up the
"killer application" that will cause everyone to
adopt ubiquitous computing technology. I think
this is the wrong goal to have in mind as a
researcher for two reasons. First, we are not
schooled in the practices to judge a market
Second, my interpretation of Weiser's vision is
not that there would be a single, most compelling
use of ubicomp technology, but rather that the
totality of the experience would cause a
significant paradigm shift in the way we as
humans perceive our relationship to interactive
computing. In short, Weiser argued for what I
would like to call a "killer existence," in which
many disparate devices would provide an array of
services in ways that seamlessly complement our
lifestyles. -- Gregory Abowd, 2001
55
Why Context-Awareness?
Physical World
Virtual World
Sensors
Computers
56
Why Infrastructure Approach?

• Sharing of data
• Collaborative filtering, large data sets
• Sharing of processing power
• Heavy-duty processing on dedicated machines
• Sharing of sensors
• Sensors in the environment
• In-place evolution of services
• New algorithms, new features
• Supports heterogeneity
• Sensors, Devices, OS, Programming Langs

57
Why Infrastructure Approach?

• Building and evaluating now can
• Provide more design space for interaction
• Let us rapidly prototype and evaluate these apps
• Find problems now
• Develop the right mechanisms
• Evolve the right policies

58
Other Related Work

• Context Toolkit
• Widgets, Interpreters, Aggregators
• ParcTab system
• User agents, Device agents, Active Maps
• Interactive Workspaces
• Event heap, Path Creation
• MUSE
• Jini services, Bayesian reasoning

59
Some Sensors

• Temperature
• Pressure
• Humidity
• Soil makeup
• Acceleration
• Noise
• Material stress

• Biometric
• Motion
• Distance
• Touch
• Location
• Orientation
• Light

60
Related WorkContext Toolkit

• Widgets, Interpreters, Aggregators
• Hardwired data flow

In/Out Board
Building Aggregator
ID to Name Interpreter
Face Recognition
Smart Card Reader
61
Related WorkParcTab System

• Will need to re-implement parts of ParcTab
  system
• Device Agents / User Agents / Active Maps

62
Related WorkInteractive Workspaces

• Event Heap
• Scaling of tuple spaces will be a problem
• Can use how events are described

63
Related WorkMUSE

• Sensor management issues
• Sensor discovery (currently via Jini)
• Power consumption
• Probabilistically modeling sensor data
• Sensor fusion (via Bayesian nets and HMMs)

64
Rough Taxonomy ofContext-Aware Apps

• Triggers
• Metadata Tagging
• Reconfiguration and Streamlining
• Input specification
• Presentation

65
Rough Taxonomy ofContext-Aware Apps

• Triggers
• On X do Y
• "Notify doctor and nearby ambulances if serious
  health problem detected" (Citris)
• "Remind me to talk to Chris about user studies
  next time I see him" (Proxy Lady)
• Metadata Tagging
• "Where was this picture taken?"
• "Find all notes taken while Mae was talking"
• Memory prosthesis (Forget-me-not)

66
Rough Taxonomy ofContext-Aware Apps

• Reconfiguration and Streamlining
• Telephone forwarding and Teleport (Active Badges)
• Turn off cell phone in theaters
• Automatically adjust brightness / volume
• Automatic file pre-caching (OceanStore)
• Select modes in multimodal interaction
• Multimedia/Bandwidth adaptation (Odyssey)

67
Rough Taxonomy ofContext-Aware Apps

• Input specification
• Send mail to people in building (Geocasting)
• Print to nearest printer (ParcTabs)
• "Find gas stations nearest me"
• Presentation
• Current location (Cyberguide)
• Idle? (Yahoo Messenger, finger)
• Currently in? (In / Out board)
• Contextual info about objects (CoolTown)
• Proximate selection (ParcTab)

68
Context-Aware Mechanisms

• Events
• Something interesting has just happened
• e.g. "Larry has entered room 306"
• Queries
• Introspect current context state (discrete)
• e.g. "Where is Larry now?"
• Filters
• Modify events and queries based on preferences
  and privacy
• e.g. "Only show others if in office or not"

69
Are Queries and Events Enough?
Query
Event
Triggers
On Event do Action
Metadata Tagging
Query for relevant context state
Reconfiguration
Query current context state before and while
running
On Event do Reconfigure
Input Specification
Query current context state to fill in the blanks
Presentation
Query current state for more information
On Event do Update Presentation
70
What I Will Not Be Working On

• Building a new service infrastructure
• Developing new sensors
• Extremely sophisticated forms of context
• Natural language processing
• User intention
• New algorithms for
• Sensor / context fusion
• Machine learning
• Power consumption
• Just use algorithms that already exist!

71
What I Will Not Be Working On

• Sophisticated preferences
• Enable us to explore this space more
• Comprehensive security and privacy mechanisms
• Will implement basic preferences, filters, and
  access control
• One subgoal is to get the defaults right
• Rigorous user evaluations