Image Demosaicing: a Systematic Survey

1
Image Demosaicing a Systematic Survey

• Xin Li1, Bahadir Gunturk2 and Lei Zhang3
• 1Lane Department of CSEE, West Virginia
  University
• 2Dept. of ECE, Louisiana State University
• 3Dept. of Computing, The Hong Kong Polytechnic
  University

2
Growth of Image Demosaicing Community
Distribution of researchers across the world
Published papers since Year 1999
3
Acknowledgement

• Yap-peng Tan at Nanyang Technological University
  (NTU), Singapore
• David Alleysson at University Pierre Mendes
  France (UPMF)
• Daniele Menon at University of Padova, Italy
• King-Hong Chung at the Hong Kong Polytechnic
  University (HKPU)
• Dmitriy Paliy at Tampere University of
  Technology, Finland
• Chung-Yen Su at National Taiwan Normal University
• Darian Muresan previously with Cornell University
  
• Keigo Hirakawa at Harvard University

4
Outline of This Talk

• Color science background
• Scientific basis of color-filter-array (CFA)
• Image demosaicing problem formulations
• Deterministic in the frequency domain
• Statistical in the spatial domain
• Categorization of existing methods
• Sequential vs. parallel reconstruction
• Performance evaluation
• Comparison results
• Two image data sets Kodak CD and IMAX HD
• Concluding remarks and open questions
• What have we learned? What lies ahead?

5
Trichromatic Color Vision
(M)
(S)
(L)
6
Biological Demosaicing Problem
Human vision system (HVS) solves this biological
demosaicing problem so well that trichromacy does
not affect spatial acuity1
1Alleysson, D. Susstrunk, S. Herault, J.,
"Linear demosaicing inspired by the human visual
system," IEEE Transactions on Image Processing,,
vol.14, no.4, pp. 439-449, April 2005
7
Computational Demosaicing Problem
S(R,G,B)
3-CCD camera
zS(zR,zG,zB)
single-CCD camera
8
Computational Demosaicing Problem (Cond)

Objective minimize the distortion between S and
S for the class of images of interests.
9
Statistical vs. Deterministic Formulation
Bayesian perspective
Spectral perspective1
Original S
zG
zR/zB
1David Alleysson et al., Frequency selection
demosaicking a review and a look ahead
10
Categorization of Existing Demosaicing Methods
H
Hinter
Hintra
sequential
parallel
Iterative methods
Luminance
Vector median filter
Chrominance
Neural network (NN) or VQ-based
(Post-processing)
11
Selected Example Sequential Demosaicing
(blue channel is processed in a similar fashion
to red channel)
luminance
chrominance
Edge-sensitive/directional interpolation, local
polynomial approximation
12
Experimental Set-up

• Eleven latest algorithms have been used in our
  comparison
• Lu Tan (LT) TIP Oct. 2003
• Alternating projection (AP) TIP Sep. 2002
• Adaptive homogeneity-directed (AHD) TIP Mar.
  2005
• Successive approximation (SA) with edge-weighted
  improvement TIP Feb. 2005TCE May 2006
• Lukacs CCA method with post-processing TCE
  2004ICIP2004
• Frequency-domain demosaicing (FD) TIP April 2005
• Directional filtering and a posteriori decision
  (DFPD) TIP Jan. 2007
• Variance of color difference (VCD) TIP Oct. 2006
• Directional Linear MMSE estimation (DLMMSE) TIP
  Dec. 2005
• Local polynomial approximation (LPA) IMA 2007
• Adaptive filtering (AF) TIP Oct. 2007

13
Performance Evaluation Protocols
KODAK test images
IMAX test images

• Objective measures PSNR values SCIELab metrics
• Subjective evaluation very limited (mainly to
  visually inspect the severity of various
  artifacts)

14
PSNR Performance Comparison on KODAK set
15
S-CIELab Measure Comparison on KODAK Set
16
Subjective Performance Comparison Examples
original LT AP AHD SA CCA
FD DFPD VCD DL LPA AF
17
PSNR Performance Comparison on IMAX set
18
Subjective Performance Comparison Examples
original LT AP AHD SA CCA
AF DFPD VCD DL LPA NEDI
19
Discussions and Perspectives

• What have we learned?
• Color-related problems are hard and our
  understanding of color demosaicing problem
  remains ad-hoc
• Many demosaicing techniques might appear
  different but essentially follow a similar
  motivation
• Kodak image set is a poor benchmark despite its
  popularity1
• What are important questions ahead?
• Establishment of an alternative benchmark data
  set for demosaicing research
• Design of new-generation CFA2 and video
  demosaicing techniques
• Exploration of its relationship to other tasks
  such as compression, denoising and forensics

1Xiaolin Wu et al., Improved color demosaicking
in weak spectral correlation
2Keigo Hirakawa and Patrick J. Wolfe
Second-generation CFA and demosaicking designs
20
Conclusion Demosaicing is never an isolated
problem Instead of paying attention to PSNR
values, it is often more fruitful to rethink
this problem under the context of electronic
imaging and ask the right question first.
21
Ad-hoc Fusion of Different Demoisaicing Methods