ETISEO Project

1
ETISEO Project
Ground Truth Video annotation
2
ETISEO - GT
For each video sequence, the evaluator will
generate the corresponding GT - Using the
Viper-GT Annotation tool (U Maryland ).
Video sequences are characterized considering
- day time day, sunrise, nightfall, night -
Weather conditions sun, cloud, rain, fog, snow
- Illumination variations none, slow or
fast - Shadows no shadows, weak shadow or
contrasted -
Enabling a graduation of difficulties from easy
to hard .
3
Ground Truth
Physical object annotations include - Bounding
Box - Type of the object Person, Vehicle,
Group, - Sub-Type as Car, Truck or Loader
for vehicle - States Static, occluded
Event annotations include - Event type (
Ontology ) - Starting time (frame), - Ending
time (frame),
4
Contextual Information
Contextual information associated to videos -
3D empty scene model Minimum information few
3D distances drawn on the image Maximum
information 3D scene model made available

• Camera calibration
• - Set of 2D and 3D points
• - Calibration matrix
• (with or without distorsion)

5
ETISEO Project
Tasks Metrics
6
Tasks evaluated
GT Metrics are designed to evaluate tasks all
along the video processing chain Task 1
Detection of physical objects, Task 2
Localisation of physical objects, Task 3
Classification of physical objects, Task 4
Tracking of physical objects, Task 5 Event
recognition.
7
ETISEO Metrics
A variety of metrics is proposed for a detailed
analysis of the algorithms - Based on
quantitative evaluations, - Applied on
large diversity of sequences.
Working documents are on www .etiseo.net -
metrics definition, - Video annotation rules.
8
ETISEO Criteria
Following criteria are applied to qualify
algorithms

• - Number of correctly detected objects in each
  frame,
• - Precision of 2D localisation (centroid,
  bounding box)
• - Objects fragmentation (splitting merging),
• Tracking persistence with partial occlusion
• (static or dynamic),
• - Objects ID persistent across the video,
• - Object classification Object recognition,
• - Number of recognised events in the video
  sequences,
• Correct scenario recognition on time
• (starting ending time)
• - Precision 3D trajectories.

9
Partners Results
Partners must submitted their results - in
suitable time, - with XML compatible format.
The Evaluator provides necessary utilities in
order to allow participants to exploit existing
software being compatible with GT.
Participants answer questionnaire associated to
algorithms results (time processing, learning
phase, parameters data set, mask, calibration).
10
ETISEO Project
Evaluation Cycles
Procedures Rules
11
Basic evaluation process

• Participant registration,
• Ask for corpus data
• trough ETISEO web-site formulary,
• Acceptance of limited usage of the videos.

• Receive video corpus,
• Process tests on its own,
• Send back results to the evaluator.

• The evaluator computes the comparison with GT,
• Send metric values to the participant.

12
3 Phases

• Definition Preparation, 2005
• - Metrics definition, tools creation
• - video sequences recording

• Test cycle for validation, 1st semester 2006
• Realisation of an entire evaluation cycle
  with participants on the test data set.

• ETISEO evaluation cycle, 2rd semester 2006
• Final ETISEO evaluation cycle.

13
3 Seminars

• 1st seminar 10-11 may 2005, Nice, France
• - Definition process ressources generation.

• 2rd seminar INRETS, 15-16 Dember,
• Villeneuve dAsc ,France
• - resources distribution,
• - starting test cycle with participants.

• 3rd seminar INRIA, end of 2006
• Sophia-Antipolis, France
• - evaluation results communication,
• - new knowlegde analysis,
• - End of ETISEO project .

14
ETISEO 2006 steps
15
ETISEO Project
Results Dissemination
16
Participants Results

• During the project results are not diffused by
  the evaluator.

- Results are transmitted by the evaluator -
to each participant, - to INRIA, the scientific
leader
- Results publication will be realised with
participants agreement.
17
ETISEO Dissemination
- The assembly of the results will be presented
during the last ETISEO seminar (end 2006).

• New knowledge on the evaluation processing,
  pertinence of the video dataset, the annotation
  and of the metrics will be discuss.

• ETISEO Ressources tools will be diffused
  contributing to evaluation good practices
  diffusion.

18
www .etiseo.net
david.cher_at_ silogic.fr Phone [phone]
57
Project coordinator Mr. David CHER
www.silogic.fr
francois.bremond_at_ sophia.inria.fr Phone
33 (0)4 92 38 76 59
Scientific leader Mr. François BREMOND
www-sop.inria.fr /orion