Attribute Multiset Grammars for Global Explanations of Activities

1
Attribute Multiset Grammars for Global
Explanations of Activities

• Dima Damen, David Hogg
• Computer Vision Group

2
Activity Recognition
3
Activity Recognition
4
Activity Recognition
5
Activity Recognition
6
Activity Recognition
7
Activity Recognition
8
Contribution

• Define global explanations using Attribute
  Multiset Grammars
• Parsing a set of detections ?recognition
• Bayesian approach to finding the best parse tree
• Two case studies

9
Activity Recognition
Event Definition
1 event
10
Activity Recognition
Event Definition
Event Threads
?
11
Activity Recognition
Event Definition
Global Explanation
12
Activity Recognition

• Uncertain detections
• Multiple definitions

13
Activity Recognition

• Intra-activity constraints
• Temporal Constraints
• Spatial Constraints
• Other Geometric Constraints
• Inter-activity constraints

14
Activity Recognition

• Complex events

15
Definition using AMG
16
Definition using AMG
17
Definition using AMG
E ? a, c c.count lt 1
D ? A, E .
18
Definition using AMG
1
1
1
2
2
2
3
3
4
2
3
1
1
1
2
2
3
4
19
Definition using AMG
1
1
1
2
2
2
3
3
4
2
3
1
1
1
2
2
3
4
20
Definition using AMG
1
1
1
2
2
2
3
3
4
1
1
1
2
3
4
2
2
2
3
21
Definition using AMG
1
1
1
2
2
2
3
3
4
22
Definition using AMG

• Attribute Multiset Grammars
• G (N, T, S, A, P)

Multiset
23
Recognition using AMG
D a1, a2, b1, c1, c2
24
Recognition using AMG
D a1, a2, b1, c1, c2
Algorithm in Paper
25
Recognition using AMG
D a1, a2, b1, c1, c2
26
Recognition using AMG
D a1, a2, b1, c1, c2
27
Recognition using AMG
Searching the space of explanations CVPR 09

• Greedy Search
• Multiple Hypotheses Tree BMVC 07
• Reversible Jump Markov Chain
• Monte Carlo

28
Case I The Bicycles Problem
29
The Bicycles Problem
30
The Bicycles AMG
31
Case II The Enter-Exit Problem
32
Global Explanation

• Global explanation

33
Global Explanation
time
34
Using Baggage Detector ECCV 08
35
Selected Features

• 1. Matching Heights

36
Selected Features

• 2. Clothing Colour

37
Selected Features

• 3. Baggage Colour

38
Selected Features

• 4. Baggage Relative Height

39
Experiment

• 12 hours
• 326 people
• 429 candidate bags
• 62 ground truth pairs

40
Results
41
Conclusion

• Defining activity using AMG
• Hierarchies of events
• Multisets
• Intra-activity constraints ? synthetic attributes
• Inter-activity constraints ? inherited attributes
• Finding the best parse tree ? Recognition
• Building BN
• Searching for MAP
• Two case studies
• Future work

42
Thank you ?