INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval

1
INAOE at ImageCLEF2007Towards Annotation based
Image Retrieval

• H. Jair Escalante, Carlos Hernández, Aurelio
  López, Heidi Marín, Manuel Montes, Eduardo
  Morales, Enrique Sucar, Luis Villaseñor
• Language Technologies Laboratory
• National Institute of Astrophysics, Optics and
  Electronics
• Tonantzintla, Mexico
• mmontesg_at_inaoep.mx
• http//ccc.inaoep.mx/mmontesg

2
Overview of the talk

• Our first participation at ImageCLEF the goal
  was to build the basic infrastructure
• Some textual and mixed strategies for image
  retrieval
• However we could do something more
• A Web based query expansion method, and
• An annotation based image retrieval approach

3
Textual and mixed strategies

• VSM IR System for textual retrieval (baseline)
• Late fusion of independent retrievers (LF)
• Intermedia feedback (IMFB)

Topic statement
TBIR
Fusion
Query
RelevantImages
CBIR
Example images
4
Some new things

• Web-based query expansion
• Original statement top-k snippets (NQE)
• Original statement top-l more repeated words
  from the top-k snippets (WQE)
• Annotation based expansion (ABE)
• Use automatic image annotation methods for
  obtaining text from images, then
• Expand documents and/or queries with automatic
  annotations, finally
• Apply some strategy for textual image retrieval

5
Basis of our idea

• Region-level annotations are generally
  complementary to manual (image-level) annotations

sky
palm
palm, sky, sand, grass, sea, clouds
clouds
Flamingo Beach Original name in Portuguese
Praia do Flamengo Flamingo Beach is considered
as one of the most beautiful beaches of Brazil
sea
sand
sand
grass
Flamingo Beach Original name in Portuguese
Praia do Flamengo Flamingo Beach is considered
as one of the most beautiful beaches of Brazil
Flamingo Beach Original name in Portuguese
Praia do Flamengo Flamingo Beach is considered
as one of the most beautiful beaches of Brazil
6
Automatic image annotation

• Assign labels (words) to regions within segmented
  images

Automatic image Annotation method
. . .
Sky
Elephant
Grass 0.6 Sky 0.2 Tree 0.1 Ground 0.1
Annotation improvement
Rock 0.5 Church 0.2 Elephant 0.2 Entrance 0.1
Grass 0.5 Tree 0.3 Ground 0.1 Jet 0.1
Grass
7
Improving the automatic annotation
Grass 0.6 Tree 0.2 rock 0.1 building 0.1
People 0.4 Tree 0.3 Mountain 0.2 Jet 0.1
Tree 0.5 Grass 0.3 Sky 0.1 Jet 0.1
Grass, Tree, Rock, Building, People, Mountain,
Jet, Sky, Church, Elephant
Church 0.3 Grass 0.3 Sky 0.2 Elephant 0.2
8
Set of labels
9
Some problems with the labels

• 2000 training annotated-regions (2)
• 98000 regions to annotate (98)
• Imbalanced training set
• Limited vocabulary

10
Annotation based query expansion
11
Annotation based document expansion
The surroundings of the Valle Francés Torres del
Paine National Park, Chile March 2002 furniture
grasspeople clouds
The volcano Tungurahua Baños, EcuadorMarch
2002 sand clouds sky mountain
12
Experimental results
Top ranked runs for each configuration
considered.
13
Visual-English run

• No textual query was used, but at the end the
  recovery was done based on textual data.
• It combines intermedia feedback and our
  annotation based expansion technique.

14
Textual vs. mixed strategies
15
Initial conclusions

• Intermedia feedback is an effective way for
  mixing visual and textual information
• Methods based on web-query expansion showed
  better performance
• Anotation based expansion is a promising way for
  expanding text using images visual content
• Annotations can be useful for image retrieval,
  though several issues should be addressed

16
Our current work

• Work on the improvement of automatic image
  annotation methods
• Investigate different (better) ways for measuring
  the semantic cohesion between labels and manual
  annotations
• Use such semantic cohesion estimates for
  improving image retrieval from annotated
  collections

17
Thanks for your attention
Language Technologies Laboratory National
Institute of Astrophysics, Optics and
Electronics Tonantzintla, México Manuel Montes y
Gómez mmontesg_at_inaoep.mx http//ccc.inaoep.mx/mmo
ntesg