Automatic Annotation of Humans in Surveillance Video

1
Automatic Annotation of Humans in Surveillance
Video

• D.M. Hansen,
• B.K. Mortensen,
• P.T. Duizer,
• J.R. Andersen
• T.B. Moeslund

Lab. of Computer Vision and Media
Technology Aalborg University Denmark
2
Case Shopping mall

• Upload data to central server
• Compare with annotations
• from different surveillance cameras
• Track/locate individual
• Inform security
• Same for missing person

• Many possible annotations
• Primary ones
• Appearance clothing hair
• Height
• Gaze direction

3
Agenda

• Detection of individuals
• Annotation
• Appearance
• Height
• Head direction
• Conclusion

4
Detection of individuals

• Background subtraction gt silhouettes
• Codebook method
• Only detect
• upright persons
• Aspect ratio
• Size
• Tracking based
• on zeroth order
• predictor

5
Annotation Appearance

• Color of upper body
• Color of lower body
• Color of hair
• Humans can (normally) separate 11 colors

6
Annotation Appearance

• Principle Find dominant color as opposed to
  average color
• Blobs Connected pixels with similar color
• Merge non-skin blobs with similar color
• Divide into body parts Red, Green, Blue

7
Annotation Appearance

• Temporal filtering most often occurence
• Results
• Often too poor a resolution to classify hair
• Skin-color clothing cant be handled
• Borderline colors need to be represented by two
  colors
• 33 sequences are tested. 5 errors (2 skin, 3
  blue/green)

8
Annotation Height

• Assume upright persons walking on a plane
• Calibrate the camera to the floor-plane

9
Annotation Height

• Ground point
• Convex hull points closest to bounding box
  corners
• Intersect with vertical (gravity) line from
  Median
• Temporal filter (Median)
• Results 17 sequences (4 persons)

10
Annotation Head direction

• Head direction ( for attention)
• 5 directions 0, /-45, /-90
• Detect head
• 4 features
• K-nearest classifier

11
Annotation Head direction

• Quantitative test 8 subjects. 2000 samples
• Recognition rate 80.1
• Three classes 0, left, right 98.5
• Errors high hair. Glasses
• Qualitative test 40 sequences. 7000 frames
• Persons walking towards the camera
• Temporal filtering
• Good recognition rate (resolution gt 10x15 pixels
  4.5m)
• Errors transitions

12
Conclusion

• Annotation
• Appearance, height, attention
• Simple and fast methods
• Relatively successful
• Limitations
• Occlusion need to separate and track individuals
  
• Resolution (face min. 10x15 pixels) use a zoom
  camera

The End!
13
Xtras
14
Detection of individuals

• Background subtraction
• Codebook method
• For each pixel
• Codeword(s) gt Codebook
• Update Gradual changes
• Update Rapid changes

15
Codebook
16
Updates

• Gradual changes
• Change the limits on I
• Rapid changes
• Create new codewords on the fly temporary
  background
• Temporal filters activate and delete the
  temporary background
• Maximum negative run-length

17
(No Transcript)
18

• 10 hours sequence (9.15 AM 7.15 PM)
• Every 1000 frame 971 frames (93 containing
  humans)
• Ground truth Hand segmentation

False rejection rate avg. 8.5
False acceptance rate avg. 0.14
19

• False acceptance rate constant over time
• False rejection rate constant over time (until it
  gets dark)

20
Setup
21
Color

• The hue-saturation space is divided into eight
  fields. The white field represent either white,
  gray or black depending on the brightness. At low
  brightness, the orange, yellow and pink fields
  represent brown.

22
Color

• A long shirt should not contribute to the lower
  body parts