Gesture Recognition in a Class Room Environment

1
Gesture Recognition in a Class Room Environment

• Michael Wallick
• CS766

2
Virtual Videography

• Place cameras in an environment
• Automatically edit video off-line
• Output should look like a professional editor

3
Our Implementation

• Looking at the classroom domain
• Recorded one semester of CS559 (Computer
  Graphics)

4
Computer Vision in Virtual Videography

• Understand what is happening on the chalkboard
• Writing on the board
• Understand what the professor is doing
• Location
• Actions

5
Chalkboard

• Partition the board into regions
• Regions are semantically related groups of
  writing
• Regions can be approximated using computer vision
• Lets treat this as a black box it just happens

6
Gesture Recognition

• Understand gestures or actions by a performer
• Generally used as an input to a computer
• Understand what the professor is doing
• Pointing
• Writing
• Reaching

7
Writing can be confused with Pointing and Reaching
8
Template Matching for G. R.

• Generate templates of known gestures
• Match an unknown frame with a template matching
  algorithm
• Sum of Squared Difference
• Cross Correlation
• Image Difference

9
Implement of Gesture Recognition

• The user selects several template images
• Pointing
• Reaching

10
Format the templates

• Separate the lecturer
• Crop the image
• Resize the images 256x256

11
Build the Recognition Mask

• Load each template into the mask
• For each on pixel, increment the mask at that
  location

12
Recognizing Gestures

• Separate the lecturer from foreground
• Crop and resize
• For every on pixel, increment the Score by
  that value in the mask
• Compute Confidence as
• (float) (Score/Mask_Total)
• Compute Confidence for all gestures

13
A Gesture Matches if Confidence is

• Under 50 but much larger than other gestures
• Over 50 and not too close to other gestures

14
Example Ground State
15
Example Pointing
16
Example Reaching
17
Mistakes

• Overall the results are good
• Sometimes individual frames are not correct

18
Solution

• For each frame, look at surrounding frames
• Label frame with gesture of the majority

19
Where to go from here

• Use the regions to
• Validate the gestures
• Determine what is being pointed at
• Incorporate the writing information with the
  gestures
• Write paper and webpage!

20
Conclusions

• We want to use gesture recognition for Virtual
  Videography
• Gestures can be used to drive camera model
• Find gestures by template matching
• For each frame, take the average around a
  region of frames to correct errors

21
Thank You!

• Questions/ Comments?