A Lightweight Image Retrieval System for Paintings

1
A Lightweight Image Retrieval System for Paintings

• T. Lombardi, S. Cha, and C. Tappert
• January 19th, 2005

2
Introduction

• Students of art history learn
• three primary skills
• Formal analysis
• Comparison
• Classification
• How can computer science
• contribute to the development
• of these skills?

Figure 1 Girl with a Pearl Earring, Jan
Vermeer, 1665
3
Working Hypothesis

• An Interactive Indexing and Image Retrieval
  System (IIR) for fine-art paintings can aid
  students in these endeavors by providing
• a mathematical summarization of an image
• a measurable basis for comparing two images
• an elementary way to classify an image relative
  to those in a database

4
Previous Work

• We synthesize the goals of two research areas
• Classification of paintings
• R. Sablatnig, P. Kammerer, and E. Zolda,
  Hierarchical Classification of Paintings Using
  Face- and Brush Stroke Models, in Proc. of the
  14th International Conference on Pattern
  Recognition (1998).
• D. Keren, Painter Identification Using Local
  Features and Naïve Bayes, in Proc. of the 16th
  International Conference on Pattern Recognition
  (2002).
• Image retrieval which aims to bridge the semantic
  gap
• J. Corridoni, A. Del Bimbo, and P. Pala,
  Retrieval of Paintings using Effects Induced by
  Color Features, in Proc. of the International
  Workshop on Content-Based Access of Image and
  Video Databases (1998).
• Can we construct a feature set that satisfies the
  objectives of both areas while providing
  analytically relevant data to students?

5
System Overview

• The system consists of two major components
• Image Database
• stores images, thumbnail images, and extracted
  features for later retrieval and analysis.
• Graphical User Interface
• provides interactive query capabilities to the
  end user

6
Database Construction

• An XML index file stores extracted features and
  control information.
• A file system stores images and thumbnail images.
• The open design of the database contributes to
  the goals of ease of use and exchange of
  information.

7
Database Construction Cont.
Figure 3 File System
Figure 2 XML Index File
8
Global Feature Extraction

• Two different kinds of features are extracted
• Palette features
• concern the set of colors in an image (color map)
• examples palette scope
• Canvas features
• concern the spatial and frequency distribution of
  colors in an image (image index)
• examples max, min, median, mean (for each color
  channel)

9
Sample Feature Set
Table 1 Sample Features used for Web Museum
Interactive Test
Feature Name Description and Notes
Max Max value of H, S, and V channels
Min Min value of H, S, and V channels
Mean Mean of H, S, and V channels
Median Median of H, S, and V channels
Standard Dev. Std of H, S, and V channels
Color Entropy Measures the frequency distribution of color
Line Count Normalized number of detected edges Sobel edge detector
Intensity Mean Arithmetic mean of values in a grayscale image
10
Example Palette Scope
Figure 4 Hallucinogenic Toreador Salvador Dali,
1970
Figure 5 Composition with Large Blue Plane, Red,
Black, Yellow, and Gray Piet Mondrian, 1921

• Palette Scope -- the total number of unique
  colors used in an image.
• We expect Dalis piece to have a higher palette
  depth than Mondrians work.

11
Example Palette Scope Cont.

• Formal definition of Palette Scope (U)
• U C/P
• Where
• CTotal of unique colors measured in RGB or HSV
  triples.
• P Total of pixels in an image.

12
Example Palette Scope Cont.
Table 2 Palette Scope statistics.
Artist Total Pixels (P) Total Colors (C) Palette Depth (U)
Mondrian 359700 2242 0.00623
Dali 165775 3899 0.02351
We see that Dali uses more of the color spectrum
than Mondrian. Palette depth is an important
feature for artist and period style
identification because many styles are defined by
color, i.e. Picassos Blue Period and fauvism.
13
Graphical User Interface

• The GUI consists of three primary windows for
• Analysis
• Comparison
• Classification

14
Analysis Window
Figure 6 The Analysis Window
15
Comparison Window
Figure 7 The Comparison Window
16
Classification Window
Figure 8 The Classification Window
17
Test Results

• Two types of tests were conducted
• Feature tests
• Feature tests focus on the accuracy of specific
  collections of features.
• Interactive tests
• Interactive tests assess the accuracy of the
  system as a whole.

18
Feature Test
Figure 9 Les Demoiselles dAvignon, Pablo
Picasso, 1907.
Figure 10 Road with Cypress and Star, Vincent
Van Gogh, 1890.
Table 3 Feature test to distinguish the work of
Picasso and Van Gogh.
Training Set Test Set Percent Correct
36 36 94
200 200 88
200 200 83
19
Initial Interactive Test

• Database of 10 works of each of the following ten
  artists
• Braque, Cezanne, De Chirico, El Greco, Gauguin,
• Modigliani, Mondrian, Picasso, Rembrandt, and Van
  
• Gogh.

Table 4 Initial Interactive Test
Training Set Testing Set Percent Correct
100 90 81
20
Interactive Test Web Museum
Table 5 Results from Web Museum Interactive Test
Artist Training Set Queries Success Percent
Aertsen 9 9 5 55.6
El Greco 10 7 4 57.1
Hopper 10 7 3 42.9
Malevich 10 11 8 72.7
Monet 10 10 6 60.0
Morisot 10 11 7 63.6
Rembrandt 10 33 25 75.8
Renoir 10 38 14 36.8
Turner 10 10 4 40.0
Velazquez 10 8 8 100.0
Overall 500 293 165 56.3
21
Evaluation ofWeb Museum Test Results

• Overall result 56.3 accuracy
• 36.3 better than blind guessing (10 guesses/50
  artists 20)
• Dissecting the classification mistakes reveals
  some intelligent mistakes
• Rembrandt is most often confused with Caravaggio,
  Ast, and Vermeer

22
Conclusions

• Simple palette and canvas features are sufficient
  for an interactive classification system
• A single feature set can serve for classification
  and image retrieval applications
• A general feature set can adequately serve for
  educational applications
• Although showing promise, we currently have a low
  confidence system

23
Future Work

• Add texture features
• Improved color features hue histograms
• Improved distance metrics modulo comparison of
  hue histograms
• Test against larger datasets