Image Interestingness

1
Image Interestingness

• Harish Katti
• Prof Mohan Kankanhalli
• Prof Chua Tat-Seng

2
Outline

• Interestingness, an aesthetic attribute
• The problem Interestingness
• Related work Pre-attentive vision and Global
  properties
• Our approach
• Results

3
Interestingness

• Properties of visual content in images belonging
  to a conceptual category which make the images
  preferred and desirable for viewing by users as
  compared to other images in the same category

4
Interestingness An aesthetic attribute

• make the images preferred and desirable for
  viewing in a semantic category

Sunrise
Matsumoto Castle
5
Outline

• Interestingness, an aesthetic attribute
• The problem Interestingness
• Related work Pre-attentive vision and Global
  properties
• Our approach
• Results

6
Aspects of interestingness !

• pre-attentive (lt 50ms) ?
• Does a quantitative measure exist ?
• Has categories ?

Cognitive Load, Time-span to understand
Colour ?
Structure Colour ?
Abstract thought ?
Symmetry ?
Source Flickr
7
Aspects of interestingness !
Increasing Cognitive Load, Time to understand ?
High Emotional Content ?
Violation of real world reasoning ?
Symmetry ?
Structure ?
Colour ?
8
Image pool
People agree well with Flickr
9
Local and Global properties
Global Image level
From Vogel et. al. 2007
Local block level
10
Approach

• Use global properties of images
• Global properties for image categories Torralba
  2003
• Pre-attentive scene classification Fei 2006
• Use of Social networking and human interaction
  with media for corpus
• Publicly accessible data from Flickr

11
Outline

• Interestingness, an aesthetic attribute
• The problem Interestingness
• Related work Pre-attentive vision and Global
  properties
• Our approach
• Results

12
Global spectral signature
Beach, coast, mountain
Logarithm of mean power spectrum (normalised
squared FFT over image) Torralba 2003
Building, highway, Indoor
13
Global properties

• Averaged spatial images
• Averaged spectral signatures
• Scene scale mean distance of observer to
  principal objects

Torralba 2003
14
In the long term everything matters !!!
Influence of blurring, scrambling, removing
colour, 4 sec presentation Vogel et. al. 2007
15
What do we see in a glance ?
Image Onset t0 secs
Mask Onset t27 to 500 msec
Time
Time
PT27 ms Mostly dark, some square things, maybe
furniture
PT40 ms Indoor shot, Large framed object, white
background
PT67 ms Interior of room, picture to right
black, table in center
Li 2006
16
Outline

• Interestingness, an aesthetic attribute
• The problem Interestingness
• Related work Pre-attentive vision and Global
  properties
• Our approach
• Results

17
The experiment

• Data from Flickr
• Interesting photos Use interaction info
  (click-throughs, recommendation, esteem)
• Control photos Ranked based on tag information
  only
• Test to validate user agreement with Flickr
  interestingness algorithm
• Forced-choice test between categories for
  pre-attentive interestingness
• Automatic clustering based on global properties
  (structure, colour)

18
Data

• 14 categories, 7 Natural, 7 man-made
• 9,137 interesting, 16,244 relevant
• Queried using a bag of words approach

19
Experiment protocol
Interesting ? First/second
64 ms
64 ms
Pre-attentive Interestingness
1000 ms
16-500 ms
16-500 ms
Time
Timeline for PT32 ms
Mask 0 ms
Noise 64 ms
Image1 32 ms
Image2 1064 ms
Noise 1100 ms
Mask 1132 ms
Question ?
20
Stimulus manipulation
Original
Only local info
Only global information
Intensity information
21
Outline

• The problem
• Interestingness, an aesthetic attribute
• Pre-attentive vision and Global properties
• Related work
• Our approach
• Results

22
Results Goodness of pre-attentive decision
Significant capability of discriminating
interestingness in pre-attentive time spans
Significant agreement of users notion of
interestingness with that of flickr in different
image modes
23
Ongoing work Categories of interestingness
Categories based on different types of image
attributes symmetry, abstraction, depth, etc
Categories based on time to discriminate
24
Interestingness is different from Relevance !
25
EXIF popularity and usage
26
Why do this ?

• Media Semantics needs definitions of aesthetic
  properties
• Are there categories ?
• Interestingness filter (classifier) Flickr, 27
  million images
• Short span discrimination gt Content presentation
  time
• Personalized agents for user-preferences in image
  browsing

27

• Still searching for that perfect sunset