Information Fusion of Flash and Non Flash Images

1
Information Fusion of Flash and Non Flash Images

• Michael Braun Georg Petschnigg
• CS223b Final Project March 14, 2002

2
Background

• Digital cameras are selling like hotcakes
• Most of them imitate traditional photography in
  look and functionality
• Can we use computer vision, better imaging
  sensors and CPU power to go beyond film?

3
Scope

• What can we do with the information contained in
  two temporally close images, one taken with flash
  the other without ?

?
4
Overview of investigation

• Flash Artifact Location
• Red-eye and shadow location
• Image Enhancement
• FlashSharpen reduces noise, improve sharpness and
  yields better overall colors
• FlashSharpen avoid shadows
• Enhanced Object Segmentation
• Two images enhance cut-out

5
Input Spectrum of Images

• Flash image is bright and sharp
• Non Flash image varies in quality
• Intrinsic params fixed, extrinsic almost same

Enhanced Object Cutout
FlashSharpen
Application
FlashSharpen with shadow correction
Locating Redeye
Blurry/unaligned
Sharp/aligned
Non Flash Image Quality
6
Locating Redeye

• Idea Use red difference in flash and non
  flash image to locate a red eye
• What color space to use?
• RGB no good, RGB correlated with luminosity
• HSV no good, no redness differences
• YCrCb, sort of ok, Cr measure pink
• CIE Lab best, a red luma-independent

7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
FlashSharpen

• Flash Images
• Deer in headlight effect, too bright
• Sharp edges and pronounced shadows
• Low light non flash images
• Noisy if sensor sensitive (ISO 1600)
• Shaky and blurry if exposure too long
• Retain a sense of natural overall lighting

11
FlashSharpen

• Idea Combine no flash images low frequency
  components with flash images high frequency
  components

From No Flash Image
From Flash Image
Magnitude
Frequency
12
Flash Image
13
Blurry No Flash Image
14
FlashSharpened
15
(No Transcript)
16
FlashSharpen

• Problem Shadows and speculars introduce unwanted
  edges and artifacts
• Idea Use computer vision to detect and avoid
  shadows in sharpening algorithm
• If the non flash image is moderatly sharp a
  shadow/specular edge is denoted by a large
  difference in high frequency components

17
Shadow appears as ghost
Shadow
18
Flash image too bright, shadow
Shadow
19
Non flash image not sharp
20
Fused image nice and sharp
21
(No Transcript)
22
Background Segmentation
23
Use of Magic Wand

• Select pixel from background

24
Use of Magic Wand

• Find adjoining values that are within tolerance
  of selected pixel
• This simple algorithm allows for variance in
  background data, but does not capture actual
  distribution

25
Extended Magic Wand

• User specifies multiplebackground pixels
• This training data carves hypercubes in YCrCb
  space
• Training algorithm can allow for creation of
  multiple hypercubes to represent background data
• Based loosely on Fuzzy ARTMAP (Carpenter,
  Grossberg, 1992)

26
Extended Magic Wand
Binary Detection Mask
27
Extended Magic Wand
Detection Mask Eroded and Blurred
28
Extended Magic Wand
Final Product Fused Image Transparency Mask
29
Use of Flash and Non-Flash Images for Improved
Segmentation
30
More Results
31
More Results
32
More Results
33
Conclusion

• Fusing information of a flash and non-flash image
  allows us to
• Increase Sharpness
• Improve overall color appearance
• Detect Flash Artifacts
• Enhance object cut-out
• Computer Vision, CPU Power and better imaging
  sensors will push digital photography beyond film