Understanding Human Behavior from Sensor Data

1
Understanding Human Behavior from Sensor Data

• Henry Kautz
• University of Washington

2
A Dream of AI

• Systems that can understand ordinary human
  experience
• Work in KR, NLP, vision, IUI, planning
• Plan recognition
• Behavior recognition
• Activity tracking
• Goals
• Intelligent user interfaces
• Step toward true AI

3
Plan Recognition, circa 1985
4
Behavior Recognition, circa 2005
5
Punch Line

• Resurgence of work in behavior understanding,
  fueled by
• Advances in probabilistic inference
• Graphical models
• Scalable inference
• KR U Bayes
• Ubiquitous sensing devices
• RFID, GPS, motes,
• Ground recognition in sensor data
• Healthcare applications
• Aging boomers fastest growing demographic

6
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Other recent research

7
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Other recent research

8
Object-Based Activity Recognition

• Activities of daily living involve the
  manipulation of many physical objects
• Kitchen stove, pans, dishes,
• Bathroom toothbrush, shampoo, towel,
• Bedroom linen, dresser, clock, clothing,
• We can recognize activities from a time-sequence
  of object touches

9
Application

• ADL (Activity of Daily Living) monitoring for the
  disabled and/or elderly
• Changes in routine often precursor to illness,
  accidents
• Human monitoring intrusive inaccurate

Image Courtesy Intel Research
10
Sensing Object Manipulation

• RFID Radio-frequency identification tags
• Small
• No batteries
• Durable
• Cheap
• Easy to tag objects
• Near future use products own tags

11
Wearable RFID Readers

• 13.56MHz reader, radio, power supply, antenna
• 12 inch range, 12-150 hour lifetime
• Objects tagged on grasping surfaces

12
Experiment ADL Form Filling

• Tagged real home with 108 tags
• 14 subjects each performed 12 of 65 activities
  (14 ADLs) in arbitrary order
• Used glove-based reader
• Given trace, recreate activities

13
Results Detecting ADLs
Activity Prior Work SHARP
Personal Appearance 92/92
Oral Hygiene 70/78
Toileting 73/73
Washing up 100/33
Appliance Use 100/75
Use of Heating 84/78
Care of clothes and linen 100/73
Making a snack 100/78
Making a drink 75/60
Use of phone 64/64
Leisure Activity 100/79
Infant Care 100/58
Medication Taking 100/93
Housework 100/82
Inferring ADLs from Interactions with
Objects Philipose, Fishkin, Perkowitz, Patterson,
Hähnel, Fox, and Kautz IEEE Pervasive Computing,
4(3), 2004
14
Experiment Morning Activities

• 10 days of data from the morning routine in an
  experimenters home
• 61 tagged objects
• 11 activities
• Often interleaved and interrupted
• Many shared objects

Use bathroom Make coffee Set table
Make oatmeal Make tea Eat breakfast
Make eggs Use telephone Clear table
Prepare OJ Take out trash
15
Baseline Individual Hidden Markov Models
68 accuracy11.8 errors per episode
16
Baseline Single Hidden Markov Model
83 accuracy9.4 errors per episode
17
Cause of Errors

• Observations were types of objects
• Spoon, plate, fork
• Typical errors confusion between activities
• Using one object repeatedly
• Using different objects of same type
• Critical distinction in many ADLs
• Eating versus setting table
• Dressing versus putting away laundry

18
Aggregate Features

• HMM with individual object observations fails
• No generalization!
• Solution add aggregation features
• Number of objects of each type used
• Requires history of current activity performance
• DBN encoding avoids explosion of HMM

19
Dynamic Bayes Net with Aggregate Features
88 accuracy6.5 errors per episode
20
Improving Robustness

• DBN fails if novel objects are used
• Solution smooth parameters over abstraction
  hierarchy of object types

21
(No Transcript)
22
Abstraction Smoothing

• Methodology
• Train on 10 days data
• Test where one activity substitutes one object
• Change in error rate
• Without smoothing 26 increase
• With smoothing 1 increase

23
Summary

• Activities of daily living can be robustly
  tracked using RFID data
• Simple, direct sensors can often replace (or
  augment) general machine vision
• Accurate probabilistic inference requires
  sequencing, aggregation, and abstraction
• Works for essentially all ADLs defined in
  healthcare literature

D. Patterson, H. Kautz, D. Fox, ISWC 2005 best
paper award
24
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Other recent research

25
Motivation Community Access for the Cognitively
Disabled
26
The Problem

• Using public transit cognitively challenging
• Learning bus routes and numbers
• Transfers
• Recovering from mistakes
• Point to point shuttle service impractical
• Slow
• Expensive
• Current GPS units hard to use
• Require extensive user input
• Error-prone near buildings, inside buses
• No help with transfers, timing

27
Solution Activity Compass

• User carries smart cell phone
• System infers transportation mode
• GPS position, velocity
• Mass transit information
• Over time system learns user model
• Important places
• Common transportation plans
• Mismatches possible mistakes
• System provides proactive help

28
Technical Challenge

• Given a data stream from a GPS unit...
• Infer the users mode of transportation, and
  places where the mode changes
• Foot, car, bus, bike,
• Bus stops, parking lots, enter buildings,
• Learn the users daily pattern of movement
• Predict the users future actions
• Detect user errors

29
Technical Approach

• Map is a directed graph G(V,E)
• Location
• Edge e
• Distance d from start of edge
• Actual (displaced) GPS reading
• Movement
• Mode foot, car, bus determines velocity range
• Change mode near bus stops parking places
• Tracking (filtering) Given some prior estimate,
• Update position mode according to motion model
• Correct according to next GPS reading

30
Dynamic Bayesian Network I
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
GPS reading
Time k
Time k-1
31
Mode Location Tracking
Measurements Projections Bus mode Car mode Foot
mode
Green Red Blue
32
Learning

• Prior knowledge general constraints on
  transportation use
• Vehicle speed range
• Bus stops
• Learning specialize model to particular user
• 30 days GPS readings
• Unlabeled data
• Learn edge transition parameters using
  expectation-maximization (EM)

33
Predictive Accuracy
How to improve predictive power?
Probability of correctly predicting the future
City Blocks
34
Transportation Routines
B
A
Workplace
Home

• Goal intended destination
• Workplace, home, friends, restaurants,
• Trip segments ltstart, end, modegt
• Home to Bus stop A on Foot
• Bus stop A to Bus stop B on Bus
• Bus stop B to workplace on Foot

35
Dynamic Bayesian Net II
gk-1
gk
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
GPS reading
Time k
Time k-1
36
Unsupervised Hierarchical Learning

• Use previous model to infer
• Goals
• locations where user stays for a long time
• Transition points
• locations with high mode transition probability
• Trip segments
• paths connecting transition points or goals
• Learn transition probabilities
• Lower levels conditioned on higher levels
• Expectation-Maximization

37
Predict Goal and Path
Predicted goal Predicted path
38
Improvement in Predictive Accuracy
39
Detecting User Errors

• Learned model represents typical correct behavior
• Model is a poor fit to user errors
• We can use this fact to detect errors!
• Cognitive Mode
• Normal model functions as before
• Error switch in prior (untrained) parameters for
  mode and edge transition

40
Dynamic Bayesian Net III
Cognitive mode normal, error
Goal
Trip segment
Transportation mode
Edge, velocity, position
Data (edge) association
GPS reading
41
Detecting User Errors
42
Status

• Major funding by NIDRR
• National Institute of Disability Rehabilitation
  Research
• Partnership with UW Dept of Rehabilitation
  Medicine Center for Technology and Disability
  Studies
• Extension to indoor navigation
• Hospitals, nursing homes, assisted care
  communities
• Wi-Fi localization
• Multi-modal interface studies
• Speech, graphics, text
• Adaptive guidance strategies

43
Papers

• Patterson, Liao, Fox, Kautz, UBICOMP 2003
• Inferring High Level Behavior from Low Level
  Sensors
• Patterson et al, UBICOMP-2004
• Opportunity Knocks a System to Provide Cognitive
  Assistance with Transportation Services
• Liao, Fox, Kautz, AAAI 2004 (Best Paper)
• Learning and Inferring Transportation Routines

44
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Other recent research

45
Task

• Learn to label a persons
• Daily activities
• working, visiting friends, traveling,
• Significant places
• work place, friends house, usual bus stop,
• Given
• Training set of labeled examples
• Wearable sensor data stream
• GPS, acceleration, ambient noise level,

46
Applications

• Activity Compass
• Automatically assign names to places
• Life Capture
• Automated diary

47
Conditional Models

• HMMs and DBNs are generative models
• Describe complete joint probability space
• For labeling tasks, conditional models are often
  simpler and more accurate
• Learn only P( label observations )
• Fewer parameters than corresponding generative
  model

48
Things to be Modeled

• Raw GPS reading (observed)
• Actual user location
• Activities (time dependent)
• Significant places (time independent)
• Soft constraints between all of the above
  (learned)

49
Conditional Random Field

• Undirected graphical model
• Feature functions defined on cliques
• Conditional probability proportional to exp(
  weighted sum of features )
• Weights learned by maximizing (pseudo) likelihood
  of training data

50
Relational Markov Network

• First-order version of conditional random field
• Features defined by feature templates
• All instances of a template have same weight
• Examples
• Time of day an activity occurs
• Place an activity occurs
• Number of places labeled Home
• Distance between adjacent user locations
• Distance between GPS reading nearest street

51
RMN Model
s1
Global soft constraints
p1
p2
Significant places
a1
a2
a3
a4
a5
Activities
g1
g2
g3
g4
g5
g6
g7
g8
g9
GPS, location
time ?
52
Significant Places

• Previous work decoupled identifying significant
  places from rest of inference
• Simple temporal threshold
• Misses places with brief activities
• RMN model integrates
• Identifying significant place
• Labeling significant places
• Labeling activities

53
Efficient Inference

• Some features are expensive to handle by general
  inference algorithms
• E.g. belief propagation, MCMC
• Can dramatically speed up inference by
  associating inference procedures with feature
  templates
• Fast Fourier transform (FFT) to compute
  counting features
• O(n log2n) versus O(2n)

54
Results Labeling

• One user, 1 week training data, 1 week testing
  data
• Number of (new) significant places correctly
  labeled 18 out of 19
• Number of activities correctly labeled 53 out
  of 61
• Number of activities correctly labeled, if
  counting features not used 44 out of 61

55
Results Finding Significant Places
56
Results Efficiency
57
Summary

• We can learn to label a users activities and
  meaningful locations using sensor data state of
  the art relational statistical models
• (Liao, Fox, Kautz, IJCAI 2005, NIPS 2006)
• Many avenues to explore
• Transfer learning
• Finer grained activities
• Structured activities
• Social groups

58
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Other recent research

59
Planning as Satisfiability

• Unless PNP, no polynomial time algorithm for SAT
• But great practical progress in recent years
• 1980 100 variable problems
• 2005 100,000 variable problems
• Can we use SAT as a engine for planning?
• 1996 competitive with state of the art
• ICAPS 2004 Planning Competition 1st prize,
  optimal STRIPS planning
• Inspired research on bounded model-checking

60
Understanding Clause Learning

• Modern SAT solvers extend Davis-Putnam backtrack
  search with clause learning
• Cache reason for each backtrack as an inferred
  clause
• Works in practice, but not clear why
• Understanding clause learning using proof
  complexity
• (Sabharwal, Beame, Kautz 2003, 2004, 2005)
• More powerful than tree-like resolution
• Separation from regular resolution
• Increase practical benefit bydomain-specific
  variable-selection heuristics
• Linear-size proofs of pebbling graphs

61
Efficient Model Counting

• SAT can a formula be satisfied?
• SAT how many ways can a formula be satisfied?
• Compact translation discrete Bayesian networks ?
  SAT
• Efficient model counting (Sang, Beame, Kautz
  2004, 2005)
• Formula caching
• Component analysis
• New branching heuristics
• Cachet fastest modelcounting algorithm

62
Learning Control Policies for Satisfiability
Solvers

• High variance in runtime distributions of SAT
  solvers
• Over different random seeds for tie-breaking in
  branching heuristics
• Sometimes infinite variance (heavy-tailed)
• Restarts can yield super-linear speedup
• Can learn optimal restart policies using features
  of instance solver trace (Ruan, Kautz,
  Horvitz 2001, 2002, 2003)

63
Other Projects

• Continuous time Bayesian networks
• (Gopalratnam, Weld, Kautz, AAAI 2005)
• Building social network models from sensor data
• Probabilistic speech act theory
• Modal Markov Logic

64
Conclusion Why Now?

• An early goal of AI was to create programs that
  could understand ordinary human experience
• This goal proved elusive
• Missing probabilistic tools
• Systems not grounded in real world
• Lacked compelling purpose
• Today we have the mathematical tools, the
  sensors, and the motivation

65
Credits

• Graduate students
• Don Patterson, Lin Liao, Ashish Sabharwal,
  Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu,
  Bill Pentney, Brian Ferris
• Colleagues
• Dieter Fox, Gaetano Borriello, Dan Weld, Matthai
  Philipose
• Funders
• NIDRR, Intel, NSF, DARPA