EE 7700

1
EE 7700

• Demosaicking Problem in Digital Cameras

2
Multi-Chip Digital Camera

• To produce a color image, at least three spectral
  components are needed at each pixel.
• One approach is to use beam-splitters and
  multiple chips.

3
Single-Chip Digital Camera

• Multi-chip approach is expensive. Precise chip
  alignment is required.
• The alternative is to use a color filter array.

Lens
Color filter array
Sensors
Scene
4
Single-Chip Digital Camera

• The missing color samples must be estimated to
  produce the full color image.
• Since a mosaic of samples are available, this
  estimation (interpolation) process is called
  demosaicking.

5
Single-Chip Digital Camera

• Images suffer from color artifacts when the
  samples are not estimated correctly.

Original image
Bilinearly interpolated from CFA-filtered samples
6
Demosaicking Approaches

• Non-Adaptive Single-Channel Interpolation
  Interpolate each color channel separately using a
  standard technique, such as nearest-neighbor
  interpolation, bilinear interpolation, etc.
• Edge-Directed Interpolation Estimate potential
  edges, avoid interpolating across the edges.

• Edge-directed interpolation
• Calculate horizontal gradient ?H G1 G2
• Calculate vertical gradient ?V G3 G4
• If ?H gt ?V,
• Gx (G3 G4)/2
• Else if ?H lt ?V,
• Gx (G1 G2)/2
• Else
• Gx (G1 G2 G3 G4)/4

3
1
2
x
4
7
Demosaicking Approaches

• Edge-Directed Interpolation Based on the
  assumption that color channels have similar
  texture, various edge detectors can be used.

• Edge-directed interpolation
• Calculate horizontal gradient ?H (R3 R7)/2
  R5
• Calculate vertical gradient ?V (R1 R9)/2
  R5
• If ?H gt ?V,
• G5 (G2 G8)/2
• Else if ?H lt ?V,
• G5 (G4 G6)/2
• Else
• G5 (G2 G8 G4 G6)/4

1
2
4
5
6
3
7
8
9
8
Demosaicking Approaches

• Constant-Hue-Based Interpolation Hue does not
  change abruptly within a small neighborhood.
• Interpolate green channel first.
• Interpolate hue (defined as either color
  differences or color ratios).
• Estimate the missing (red/blue) from the
  interpolated hue.

Interpolate
Interpolated Red
Red
Interpolate
Green
9
Demosaicking Approaches

• Edge-Directed Interpolation of Hue It is a
  combination of edge-directed interpolation and
  constant-hue-based interpolation. Hue is
  interpolated as in constant-hue-based
  interpolation approach, but this time, hue is
  interpolated based on the edge directions (as in
  the edge-directed interpolation algorithm).

10
Demosaicking Approaches

• Using Laplacian For Enhancement Use the
  second-order gradients of red/blue channels to
  enhance green channel.

• Calculate horizontal gradient ?H G4 G6
  R5 R3 R5 R7
• Calculate vertical gradient ?V G2 G8 R5
  R1 R5 R9
• If ?H gt ?V,
• G5 (G2 G8)/2 (R5 R1 R5 R9)/4
• Else if ?H lt ?V,
• G5 (G4 G6)/2 (R5 R3 R5 R7)/4
• Else
• G5 (G2 G8 G4 G6)/4 (R5 R1 R5 R9
  R5 R3 R5 R7)/8

1
2
4
5
6
7
3
8
9
11
Aliasing
Frequency spectrum of an image
After CFA sampling
Green channel
Red/Blue channel
12
Demosaicking Approach

• Alias Cancelling Based on the assumption that
  red, green, and blue channels have similar
  frequency components, the high-frequency
  components of red and blue channels are replaced
  by the high-frequency components of green
  channel.

Red/Blue channel
13
Experiment
HL
HL
HL
Full Red/Green/Blue channels
LL
LL
LL
Subband decomposition
HH
LH
LH
LH
CFA Sampling
Interpolate
HL
HL
HL
LL
LL
LL
Subband decomposition
HH
LH
LH
LH
14
Constraint Sets

• Detail Constraint Set Detail subbands of the
  red and blue channels must be similar to the
  detail subbands of the green channel.

HL
HL
HH
HH
LH
LH
15
Constraint Sets

• Observation Constraint Set Interpolated
  channels must be consistent with the observed
  data.

Sensors
CFA
16
Projection Operations

• Projection onto the Detail Constraint Set
• Decompose the color channels.
• Update the detail subbands of red and blue
  channels.

HL
HH
LH

• Apply synthesis filters to reconstruct back the
  channels.

17
Projection Operations

• Projection onto the Observation Constraint Set
• Insert the observed data to their corresponding
  positions.

Sensors
CFA
18
Alternating Projections Algorithm
Samples of color channels
Initial interpolation
Projection onto the detail constraint set
Projection onto the observation constraint set
Insert the observed data
Update
Iteration
19
Results
Original
Hibbard 1995
Laroche and Prescott 1994
Hamilton and Adams 1997
Kimmel 1999
Gunturk 2002
20
Results
Laroche and Prescott 1994
Hibbard 1995
Original
Hamilton and Adams 1997
Gunturk 2002
Kimmel 1999
21
Previous Methods
Gunturk02
Gunturk et al, Demosaicking Color Filter Array
Interpolation in Single-Chip Digital Cameras, to
appear in IEEE Signal Processing Magazine.
22
References

• Gunturk02 Gunturk et al, Color Plane
  Interpolation Using Alternating Projections,
  IEEE Trans. Image Processing, 2002.
• Hibbard 1995 R. H. Hibbard, Apparatus and
  method for adaptively interpolating a full color
  image utilizing luminance gradients, U.S. Patent
  5,382,976, January, 1995.
• Laroche and Prescott 1994 C. A. Laroche and M.
  A. Prescott, Apparatus and method for adaptively
  interpolating a full color image utilizing
  chrominance gradients, U.S. Patent 5,373,322,
  December, 1994.
• Hamilton and Adams 1997 J. F. Hamilton Jr. and
  J. E. Adams, Adaptive color plane interpolation
  in single sensor color electronic camera, U.S.
  Patent 5,629,734, May, 1997.
• Kimmel 1999 R. Kimmel, Demosaicing Image
  reconstruction from CCD samples, IEEE Trans.
  Image Processing, vol. 8, pp. 1221-1228, 1999.