Recognizing Activities of Daily Living from Sensor Data

1
Recognizing Activities of Daily Livingfrom
Sensor Data

• Henry Kautz
• Department of Computer Science
• University of Rochester

2
Activity Recognition

• Much recent interest in recognizing human
  activity from heterogeneous sensor data
• Motion sensors
• GPS
• RFID
• Video
• Compelling applications
• Military / security operations (e.g. ASSIST)
• Smart homes offices

3
Gathering data on indoor activities
4
Interpreting RFID Data (using Switching HMM)
5
Gathering Multi-view Video
6
Interpreting Video
Computing scene statistics
Ai activity Oi object Si scene statistic Di
object statistics Ri RFID label (for training)
Computing object statistics
7
Gathering data on outdoor activities

• Raw GPS

8
Discovering significant places

• Conditional Random Field

9
Predicting transportation goals
Dynamic Bayesian Network
10
Issue

• Previous work on activity recognition has used a
  wide variety of probabilistic models for
  different tasks and kinds of data
• HMMs, DBNs, CRFs,
• Background knowledge is implicitly encoded in the
  structure of the models
• E.g. Relation between transportation goals,
  plans, actions
• Increasingly difficult to scale integrate

11
Markov Logic

• Markov logic will provide common modeling
  language inference tools, enabling
• Easier integration of multiple sensors
• Easier generalization
• From one activity at a time to multiple ongoing
  activities
• From one individual to multiple individuals
• Easier modification of background knowledge
• Add / modify library of plans and goals

12
Example Scenario

• John goes into his kitchen (video)
• He takes out a jug from the refrigerator, and a
  bowl from the cabinet (RFID)
• He leaves his apartment, and walks to a
  convenience store (GPS)
• He returns carrying a box (video)
• He pours the box into the bowl (accelerometer)
  and the contents of the jug (accelerometer
  RFID)
• Why did John leave the apartment? What did he
  do?

13
UR Contributions to MURI Scenario Development
Data Collection

• Develop set of activity recognition scenarios of
  increasing complexity
• Activities in the home
• Outdoor activities
• Enact and gather sensor data
• Heterogeneous GPS, RFID, video, motion,
• Intermittent and noisy
• Make dataset available to team
• Including feature sequences extracted from video
  and acceleration data
• Ground truth
• 1st data set mid-Year One, then ongoing

14
UR Contributions to MURI Unified ML Model of
Daily Activities

• Recast our previous work on recognition using
  HMMs, DBNs, CRFs in Markov Logic
• Integrate and generalize earlier results
• Year One
• HMM ? ML
• Generalize to multiple ongoing activities
• Handle novel observations using similarity
• Representing actions, intentions, and goals
• Extend ML to include modal operators
• Distinguish beliefs of observer from beliefs of
  subject
• Ability to model imperfect agents, whose plans
  are flawed

15
From HMMs to ML

• Hidden Markov models describe the world as
  probabilistic state machine
• ML encoding of HMM can be relaxed to allow
  subject to be in multiple states (multiple
  activities) by making unique state constraint
  soft

16
From HMMs to ML

• Novel observations can be handled by applying
  background knowledge about similarity

17
Modal Operators

• Most previous work on probabilistic activity
  recognition does not distinguish
• What system infers is true about the world
• What the subject believes is true about the world
• What the system predicts will happen
• What the subject intends to happen
• Modal operators relate agents to attitudes
• Bel( John, contains(jug, gasoline) )
• But system may know jug is empty
• Goal( John, ignite(jug) )
• Knowledge of subjects goal can drive cooperative
  system to help subject, or antagonistic system to
  block user

18
Semantic Inference

• Modal operators do not work like ordinary
  predicates or logical connectives
• Modal proof theory is hard to automate
• However
• Modal operators have well-understood possible
  world semantics
• Agent believes P in possible world W iff P is
  true in all worlds W such that reachable(W,W)
• MLs inference engine works at the semantic level
  (direct search over possible worlds)
• Promising approach semantic inference for modal
  constructs in ML
• Explicitly model reachability relationships for
  each attitude and agent

19
Idea

• Alchemy searches over models (truth assignments)
• Modal formulas are evaluated over structures
• Structure set of models and accessibility
  relationships over the models
• Structures are too big to explicitly search
• Modify Alchemy to search over samples drawn from
  structures