ETISEO

1
ETISEO

• François BREMOND
• ORION Team, INRIA Sophia Antipolis, France

2
Fair Evaluation

• Unbiased and transparent evaluation protocol
• Large participation
• Meaningful evaluation

3
Tasks evaluated
GT Metrics are designed to evaluate tasks all
along the video processing chain Task 1
Detection of physical objects, Task 2
Localisation of physical objects, Task 3
Classification of physical objects, Task 4
Tracking of physical objects, Task 5 Event
recognition.
4
Matching Computation

• To evaluate the matching between a candidate
  result and a reference data, we may use following
  distances
• D1-The Dice coefficient Twice the shared,
  divided by the sum of both intervals
  2card(RD?C) / (card(RD) card(C)).
• D2-The overlapping card(RD?C) / card(RD).
• D3-Bertozzi and al. metric (card(RD?C))2 /
  (card(RD) card(C)).
• D4-The maximum deviation of the candidate object
  or target according to the shared frame span Max
  card(C\RD) / card(C), card(RD\C) / card(RD) .

RD
C
5
metrics (1)
T1- DETECTION OF PHYSICAL OBJECTS OF
INTEREST C1.1 Number of physical objects C1.2
Number of physical objects using their bounding
box

• T2- LOCALISATION OF PHYSICAL OBJECTS OF INTEREST
• C2.1 Physical objects area (pixel comparison
  based on BB)
• C2.2 Physical object area fragmentation
  (splitting)
• C2.3 Physical object area integration (merge)
• C2.4 Physical objects localisation
• 2D and 3D
• Centroïd or bottom centre point of BB

6
metrics (2)

• T3- TRACKING OF PHYSICAL OBJECTS OF INTEREST
• C3.1 Frame-To-Frame Tracking Link between two
  frames
• C3.2 Number of object being tracked during time
• C3.3 Detection time evaluation
• C3.4 Physical object ID fragmentation
• C3.5 Physical object ID confusion criterion
• C3.6 Physical object 2D trajectory
• C3.7 Physical object 3D trajectory

T4- CLASSIFICATION OF PHYSICAL OBJECTS OF
INTEREST C4.1 Object Type over the
sequence C4.2 Object classification per
type C4.3 Time Percentage Good
Classification card RD?C, Type(C) Type(RD)
/ card(RD?C)
T5- EVENT RECOGNITION C5.1 Number of Events
recognized over the sequence C5.2 Scenario
parameters
7
Metric Evaluation

• Distance for matching groundtruth and algorithms
  results
• Similar measures D1, D2, D3, D4.
• Few main metrics measure general trends
• Discriminant and meaningful
• Detection M1.2.1 CNumberObjectsBoundingBox
• Localization M2.4.3 CCentroid2DLocalisationPix.
• Tracking M3.3.1 CtrackingTime
• Object Classification M4.1.3 CobjectTypeOverSeque
  nceBBoxID
• Event Recognition M5.1.2 CNumberNamedEvents

8
Metric Evaluation (contd)

• Secondary metrics
• Complementary information
• Pixel-based (M2.1.1) versus object-based (M1.2.1)
  metrics
• Potential algorithm errors.
• Example M3.3.1 complemented (eg., about
  stability) by M3.2.1, M3.4.1 and M3.5.1.
• Non-informative Metrics
• Add noise to the evaluation or non-discriminative
• Example M1.1.1 CNumberObjects gives the object
  number per frame without position information.
• The same for M4.1.1 and M5.1.1.

9
Evaluation on ETI-VS2-BE-19-C1
10
Global Results Video

• Remarks
• For similar scenes, very dissimilar results!
• For different scenes, results can spread over a
  large range or concentrate in a narrow range.

11
Detection of Physical Objects (ETI-VS2-BE-19-C1.x
ml)
M1.1.1 NumberObjects
M1.2.1 NumberObjectsBoundingBoxD1
12
Detection of Physical Objects (ETI-VS2-BE-19-C1.x
ml)
M1.2.1 NumberObjectsBoundingBoxD1
M2.1.1 ObjectsArea
13
Detection of Physical Objects (ETI-VS2-BE-19-C1.x
ml)
M2.2.1 SplittingD5
M2.3.1 MergingD2
14
Summary on Detection of Physical Objects

• Main metric measures
• Detection M1.2.1 CNumberObjectsBoundingBox
• Problems static objects, contextual objects,
  background, masks
• Advantages objects vs pixels, large objects and
  bounding boxes
• Secondary metrics
• M2.1.1 (area) indication on the precision and
  handling shadows
• Split/Merge measures (M2.2.1, M2.3.1)
• Advantage indicate potential merge
• Inconvenients threshold-dependent, non-detected
  objects not taken into account

15
Localisation of Physical Objects
16
Localisation of Physical Objects
(ETI-VS2-BE-19-C1.xml)
M2.3.1 MergingD2
M2.4.3 Centroid2DLocalisationPixD1
17
Localisation of Physical Objects
(ETI-VS2-BE-19-C1.xml)
M2.4.1 Centroid2DLocalisationD1
M2.4.3 Centroid2DLocalisationPixD1
18
Localisation of Physical Objects
(ETI-VS2-BE-19-C1.xml)
M2.4.2. Centroid3DLocalisationD1
19
Summary on Localisation of Physical Objects

• M2.4.1, M2.4.2, M2.4.3, main metrics
• Problems low utilisation of 3D info and
  calibration
• Good performance good precision on reliable TP
  (handling shadow and merge)
• Advantages complementary to the Detection
  normalised, pixel or meter metrics

20
Tracking of Physical Objects
21
Tracking of Physical Objects (ETI-VS2-BE-19-C1.xm
l)
M3.2.1 NumberObjectTrackedD1
M3.3.1 TrackingTime
22
Tracking of Physical Objects (ETI-VS2-BE-19-C1.xm
l)
M3.4.1 PhysicalObjectIdFragmentation
M3.5.1 PhysicalObjectIdConfusion
23
Tracking of Physical Objects (ETI-VS2-BE-19-C1.xm
l)
M3.6.1 PhysicalObject2DTrajectories
M2.4.1 Centroid2DLocalisationD1
24
Summary on Tracking of Physical Objects

• M3.3.1, main metric
• Problems propagation of detection errors
• Advantages good global overview
• M3.2.1, secondary metric
• Good performance consistent TP over time for few
  TPs
• Problems not taking into account of complete FN
• Fragmentation/confusion (M3.4.1, M3.5.1)
• Advantage indicate potential ID switching
• Inconvenients not discriminative favoring
  under-detection (few IDs) over-detection
  (multiple IDs)

25
Object Classification
26
Object Classification (ETI-VS2-BE-19-C1.xml)
M4.1.1 ObjectTypeOverSequence
M4.1.1b ObjectTypeOverSequenceBoundingBoxD1
Subtype
27
Object Classification (ETI-VS2-BE-19-C1.xml)
M4.1.3 ObjectTypeOverSequenceBoundingBoxIdD1
M4.1.2 ObjectTypeOverSequenceBoundingBoxD1
28
Object Classification
29
Object Classification
30
Summary on Object Classification

• M4.1.2, M4.1.3, same main metrics
• Problems low classification of subtypes (doors,
  bikes, bags), favoring a few good quality TPs.
• Advantage reliable.
• M4.1.1 (without BBox)
• Inconvenients wrong evaluation result in case of
  double errors (classified noise and FN)
• Advantage indicate potential double errors.

31
Event Recognition
32
Event Recognition (ETI-VS2-BE-19-C1.xml)
33
Event Recognition (ETI-VS2-BE-19-C1.xml)
34
Event Recognition (ETI-VS2-BE-19-C1.xml)
M5.1.1 NumberEvents
M5.1.2 NumberNamedEventsD1
35
Summary on Event Recognition

• M5.1.2 (with time), main metrics
• Problems lack of understanding of ground truth
  definition
• Advantages good global overview per scenario
  type.
• M5.1.1, secondary metric
• Problems not taking into account of occurrence
  time

36
Evaluation Results
37
Evaluation on ETI-VS2-BE-19-C4
38
Detection of Physical Objects (ETI-VS2-BE-19-C4.x
ml)
M1.2.1 NumberObjectsBoundingBoxD1
M2.1.1 ObjectsArea
39
Tracking of Physical Objects (ETI-VS2-BE-19-C4.xm
l)

• M3.2.1 NumberObjectTrackedD1

M3.3.1.D1 TrackingTime
40
Event Recognition (ETI-VS2-BE-19-C4.xml)

• M5.1.2 NumberNamedEventsD1

M5.1.1 NumberEvents
41
Evaluation Results
42
Evaluation on ETI-VS2-MO-1-C1
43
Detection of Physical Objects (ETI-VS2-MO-1-C1.xm
l)
M1.2.1 NumberObjectsBoundingBoxD1
M2.1.1 ObjectsArea
44
Tracking of Physical Objects (ETI-VS2-MO-1-C1.xml
)

• M3.2.1 NumberObjectTrackedD1

M3.3.1.D1 TrackingTime
45
Event Recognition (ETI-VS2-MO-1-C1.xml)

• M5.1.2 NumberNamedEventsD1

M5.1.1 NumberEvents
46
Evaluation Results
47
Evaluation on ETI-VS2-RD-6-C7
48
Detection of Physical Objects
(ETI-VS2-RD-6-C7.xml)
M1.2.1 NumberObjectsBoundingBoxD1
M2.1.1 ObjectsArea
49
Detection of Physical Objects Reference Data
Filtering (ETI-VS2-RD-6-C7.xml M1.2.1 -
NumberObjectsBoundingBoxD1)
No filtering
With filtering
50
Detection of Physical Objects Reference Data
Filtering (ETI-VS2-RD-6-C7.xml M2.1.1 -
ObjectsArea)
No filtering
With filtering
51
Tracking of Physical Objects (ETI-VS2-RD-6-C7.xml
)

• M3.2.1 NumberObjectTrackedD1

M3.3.1.D1 TrackingTime
52
Tracking of Physical Objects Reference Data
Filtering (ETI-VS2-RD-6-C7.xml M3.2.1 -
NumberObjectTrackedD1)
No filtering
With filtering
53
Tracking of Physical Objects Reference Data
Filtering (ETI-VS2-RD-6-C7.xml M3.3.1.D1 -
TrackingTime)
No filtering
With filtering
54
Event Recognition (ETI-VS2-RD-6-C7.xml)

• M5.1.2 NumberNamedEventsD1

M5.1.1 NumberEvents
55
Event Recognition (ETI-VS2-RD-6-C7)
56
Evaluation Results
57
Evaluation on ETI-VS2-RD-10-C4
58
Detection of Physical Objects
(ETI-VS2-RD-10-C4.xml)
M2.1.1 ObjectsArea
M1.2.1 NumberObjectsBoundingBoxD1
59
Tracking of Physical Objects (ETI-VS2-RD-10-C4.xm
l)

• M3.2.1 NumberObjectTrackedD1

M3.3.1.D1 TrackingTime
60
Event Recognition (ETI-VS2-RD-10-C4.xml)

• M5.1.2 NumberNamedEventsD1

M5.1.1 NumberEvents
61
Evaluation Results
62
Evaluation on ETI-VS2-AP-11-C7
63
Detection of Physical Objects
(ETI-VS2-AP-11-C7.xml)
M1.2.1 NumberObjectsBoundingBoxD1
M2.1.1 ObjectsArea
64
Detection of Physical Objects Reference Data
Filtering (ETI-VS2-AP-11-C7.xml M1.2.1 -
NumberObjectsBoundingBoxD1)
No filtering
With filtering
65
Detection of Physical Objects Reference Data
Filtering (ETI-VS2-AP-11-C7.xml M2.1.1 -
ObjectsArea)
No filtering
With filtering
66
Tracking of Physical Objects (ETI-VS2-AP-11-C7.xm
l)

• M3.2.1 NumberObjectTrackedD1

M3.3.1.D1 TrackingTime
67
Tracking of Physical Objects Reference Data
Filtering (ETI-VS2-AP-11-C7.xml M3.2.1 -
NumberObjectTrackedD1)
No filtering
With filtering
68
Tracking of Physical Objects Reference Data
Filtering (ETI-VS2-AP-11-C7.xml M3.3.1.D1 -
TrackingTime)
No filtering
With filtering
69
Event Recognition (ETI-VS2-AP-11-C7.xml)
M5.1.1 NumberEvents

• M5.1.2 NumberNamedEventsD1

70
Event Recognition (ETI-VS2-AP-11-C7.xml)

• M5.1.2 NumberNamedEventsD1

M5.1.1 NumberEvents
71
Evaluation Results
72
Understanding versus Competition

• ETISEO Goal
• Not a competition nor benchmarking
• Emphasis on gaining insight into video analysis
  algorithms
• Better understanding of evaluation methodology
• Why? ETISEO limitations
• Algorithm results depend on time and manpower
  (parameter tuning),
• format understanding (XML), objective definition
  (ground truth), and algorithm capacities (static,
  occluded, portable and contextual objects)
• previous similar experiences,
• number of processed videos, frame rate, start
  frame
• Metrics and parameters (split/merge)
• learning stage required or not.

73
Understanding versus Competition (contd)

• Warmest thanks to the 16 teams
• 8 teams achieved high quality results
• 9 teams performed event recognition
• 10 teams produced results on all priority
  sequences
• Special thanks to teams 1, 8, 12, 14 and 28
• Stable and high-quality results on a large video
  set
• More evaluation results

74
Conclusions

• Good performance comparison per video automatic,
  reliable, consistent metrics.
• A few insights into video surveillance
  algorithms. For example,
• Shadows
• merge
• A few limitations
• Lack of understanding of the evaluation rules
  (output XML, time-stamp)
• Data subjectivity video, background, masks
• Metrics and evaluation parameters
• Future improvements flexible evaluation tool
• Filters for reference data
• Selection of metrics and parameters
• Selection of videos