Learning sparse representations to restore, classify, and sense images and videos

1
Learning sparse representations to restore,
classify, and sense images and videos

• Guillermo Sapiro
• University of Minnesota

Supported by NSF, NGA, NIH, ONR, DARPA, ARO,
McKnight Foundation
2

• Ramirez

Martin Duarte
Lecumberry
Rodriguez
3
Overview

• Introduction
• Denoising, Demosaicing, Inpainting
• Mairal, Elad, Sapiro, IEEE-TIP, January 2008
• Learn multiscale dictionaries
• Mairal, Elad, Sapiro, SIAM-MMS, April 2008
• Sparsity Self-similarity
• Mairal, Bach, Ponce, Sapiro, Zisserman,
  pre-print.
• Incoherent dictionaries and universal coding
• Ramirez, Lecumberry, Sapiro, June 2009, pre-print
• Learning to classify
• Mairal, Bach, Ponce, Sapiro, Zisserman, CVPR
  2008, NIPS 2008
• Rodriguez and Sapiro, pre-print, 2008.
• Learning to sense sparse signals
• Duarte and Sapiro, pre-print, May 2008, IEEE-TIP
  to appear

4
Introduction I
Sparse and Redundant Representations Webster
Dictionary Of few and scattered elements
5
Restoration by Energy Minimization
Restoration/representation algorithms are often
related to the minimization of an energy function
of the form
y Given measurements x Unknown to be
recovered

• Bayesian type of approach
• What is the prior? What is the image model?

6
The Sparseland Model for Images
M?
7
What Should the Dictionary D Be?
8
Introduction II
Dictionary Learning
9
Measure of Quality for D
Field Olshausen (96) Engan et. al.
(99) Lewicki Sejnowski (00) Cotter et. al.
(03) Gribonval et. al. (04) Aharon, Elad,
Bruckstein (04) Aharon, Elad, Bruckstein
(05) Ng et al. (07) Mairal, Sapiro, Elad (08)
10
The KSVD Algorithm General
Aharon, Elad, Bruckstein (04)
11
Show me the pictures
12
Change the Metric in the OMP
13
Non-uniform noise
14
Example Non-uniform noise
15
Example Inpainting
16
Example Demoisaic
17
Example Inpainting
18
Not enough fun yet?
Multiscale Dictionaries
19
Learned multiscale dictionary
20
(No Transcript)
21
Color multiscale dictionaries
22
Example
23
Video inpainting
24
Extending the Models
25
Universal Coding and Incoherent Dictionaries

• Consistent
• Improved generalization properties
• Improved active set computation
• Improved coding speed
• Improved reconstruction
• See poster by Ramirez and Lecumberry

26
Sparsity Self-similarityGroup Sparsity

• Combine the two of the most successful models for
  images
• Mairal, Bach. Ponce, Sapiro, Zisserman,
  pre-print, 2009

27
Learning to Classify
28
Global Dictionary
29
Barbara
30
Boat
31
Digits
32
Which dictionary? How to learn them?

• Multiple reconstructive dictionary? (Payre)
• Single reconstructive dictionary? (Ng et al,
  LeCunn et al.)
• Dictionaries for classification!
• See also Winn et al., Holub et al., Lasserre et
  al., Hinton et al. for joint discriminative/gener
  ative probabilistic approaches

33
Learning multiple reconstructive and
discriminative dictionaries
With J. Mairal, F. Bach, J. Ponce, and A.
Zisserman, CVPR 08, NIPS 08
34
Texture classification
35
Semi-supervised detection learning
36
Learning a Single Discriminative and
Reconstructive Dictionary

• Exploit the representation coefficients for
  classification
• Include this in the optimization
• Class supervised simultaneous OMP

With F. Rodriguez
37
Digits images Robust to noise and occlusions
38
Supervised Dictionary Learning
With J. Mairal, F. Bach, J. Ponce, and A.
Zisserman, NIPS 08
39
Learning to Sense Sparse Images
40
Motivation

• Compressed sensing (Candes Tao, Donoho, et al.)
• Sparsity
• Random sampling
• Universality
• Stability
• Shall the sensing be adapted to the data type?
• Yes! (Elad, Peyre, Weiss et al., Applebaum et
  al, this talk).
• Shall the sensing and dictionary be learned
  simultaneously?

41
Some formulas.
RIP (Identity Gramm Matrix)
42
Design the dictionary and sensing together
43
Just Believe the Pictures
44
Just Believe the Pictures
45
Just Believe the Pictures
46
Conclusions

• State-of-the-art denoising results for still
  (shared with Dabov et al.) and video
• General
• Vectorial and multiscale learned dictionaries
• Dictionaries with internal structure
• Dictionary learning for classification
• See also Szlam and Sapiro, ICML 2009
• See also Carin et al, ICML 2009
• Dictionary learning for sensing
• A lot of work still to be done!

47
Please do not use the wrong dictionaries

• 12 M pixel image
• 7 million patches
• LARSonline learning
• 8 minutes
• Mairal, Bach, Ponce, Sapiro, ICML 2009

48
(No Transcript)