Camera Notes

1
Camera Notes

• Xian-Sheng Hua, Shipeng Li and Hong-Jiang Zhang
• Microsoft Research Asia

ICME 2005 Amsterdam Netherlands July 7 2005
2
Motivation

• Taking notes is frequent in daily lives.
• typically by pens
• Other methods for taking notes
• Digital ink
• Typing
• Camera (photo and video)
• Audio recorder

3
Motivation

• However
• We frequently take, but seldom use
• The main reason
• Difficult to retrieve, though easy to take
• A management and searching system is desirable
• Collecting
• Classification
• Browsing / Searching
• Quality enhancement
• Exporting

4
Outline

• System Overview
• Camera Note Classification and Grouping
• Quality Enhancement and Adjustment
• Camera Note Management System
• Experiments Conclusion

5
System Overview Camera Notes

• Notes Photos and video clips

6
Camera Note Classification

• Two steps
• Note / Non-Note classification
• Separate notes from normal media collection
• Type classification
• Maps, documents, slides, whiteboards, bulletins,

7
Note / Non-Note classification

• Separate notes from normal media collection
• They are generally mixed
• It is time-consuming for manually classification
• It is observed that for notes photos/clips
  typically
• the edge density distribution is more uniform
• the background are locally near monotony
• color distribution is not balanced (only have
  several dominant colors)
• no object motion and camera motion is mild

8
Note / Non-Note classification

• Features we used
• Edge Intensity Distribution n21 dim
• Background Intensity Distribution n21 dim
• Color Distribution 256 dim
• Motion Intensity (for video clips) 1 dim
• Typical SVM is applied for training and
  classification

9
Note Type classification

• Except for the four features we also use
• Edge Orientation Distribution 36 dim
• Text Ratio 1 dim
• Average Text Height 1 dim
• Text Height Variance 1 dim
• Classification method
• Five one-versus-rest SVM classifiers
• According to the largest positive distance from
  the five classifiers

10
Grouping and Linking

• Group notes for the same item
• By timestamp and similarity
• Link to normal photos and/or video clips
• Also by timestamp

11
Quality Adjustment / Enhancement

• Automatically / Semi-automatically / Manually
• Color/Brightness/Contrast Correction
• Shape correction
• Mosaic

12
Feature Extraction

• Feature we may obtain
• Histogram
• Dominant Color
• Text (by OCR)
• User input
• Keywords (may be input by users)
• Voice description

13
Camera Notes System

• XML is applied to describe the note library

14
Camera Notes System

• XML is applied to describe the note library
• Support the following functionalities
• Note Importing
• Sorting
• Browsing
• Editing
• Searching
• Exporting

15
Experiments for note classification

• Data set
• 5h video (895 shots)
• 49 notes (shots)
• 1300 photos (by DCs and camera phones)
• 1072 notes (photos)
• Random choose half for training and half for
  testing
• And then switch

16
Experiments for note classification
17
Experiments for note classification

• Misclassified samples Correct ? Classified

Map ? Document
Slides ? Whiteboard
Bulletin ? Document
18
Conclusion

• An camera notes management system
• based on image and video content analysis
• This system enables efficient
• importing, indexing, browsing, editing, searching
  and exporting

19
Future Works

• To design more and better automatic and
  semi-automatic quality enhancement/adjustment
  schemes.
• To improve the classification accuracy.
• To support more note types.
• To provide better user interface.
• To enable more intuitive, convenient and accurate
  note searching.
• To support exporting templates enabling flexible
  and personalized outputs.

20
Thank You