Image, Video And Multimedia Systems Laboratory

1

• Image, Video And Multimedia Systems Laboratory
• Background
• http//www.image.ntua.gr

2
Automatic Image Annotation
Input image
Automatic segmentation
Desired result
3
Image Annotator Tool

• Tool forGround truth constructionSemi-automatic
  image annotation

• Support ofAutomatic segmentationManual, user
  driven region mergingExport of segmentation
  masks and textual annotation

4
Visual Descriptor Ontology

• MPEG-7(XML Schema) defines visual descriptors by
  specifying their components
• In VDO (RDFS), descriptors are defined through
  relations with their components
• Descriptors related to higher level concepts
  through inference rules
• Rules define spatio-temporal constraints

5
Knowledge-Assisted Analysis Tool
Developed in collaboration with CERTH-ITI
6
KAA Results

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltKAAgt
• ltSpatialDecomposition id"KaaMask"gt
• ltRegiongt
• ltRegionNumbergt0lt/RegionNumbergt
• ltConceptgtSealt/Conceptgt
• ltConfidencegt0.81172lt/Confidencegt
• lt/Regiongt
• ltRegiongt
• ltRegionNumbergt1lt/RegionNumbergt
• ltConceptgtPersonlt/Conceptgt
• ltConfidencegt0.948059lt/Confidencegt
• lt/Regiongt
• ltRegiongt
• ltRegionNumbergt2lt/RegionNumbergt
• ltConceptgtSealt/Conceptgt
• ltConfidencegt0.80658lt/Confidencegt
• lt/Regiongt
• ltRegiongt

7
Semantic Segmentation

• Approach
• Graph-based representation of images
• Semantic vs Syntactic regions are assigned fuzzy
  set of labels instead of low-level features
• Modification of traditional segmentation
  algorithms to operate on labelled regions
• Simultaneous image segmentation and region
  labeling
• Target
• Solve oversegmentation problems
• Assign labels with confidence values to regions
• Link labels with concepts existing in ontologies

8
Sea is oversegmented
Semantic Segmentation
People have been incorrectly merged with the sand
Region is assigned to a fuzzy set of labels
rock/0.89,sand/0.46
RSST segmentation
Sea segments are merged correctly
Semantic RSST segmentation
9
Visual Attention Classification

• Generate visual saliency maps
• Detect foreground / background
• Select most representative regions for
  classification, based on saliency

Lower Classification error