Deblurring

1
Deblurring Deconvolution

• Lecture 10

2
Admin

• Assignment 3 due
• Last lecture
• Move to Friday?
• Projects
• Come and see me

3
Different types of blur

• Camera shake
• User moving hands
• Scene motion
• Objects in the scene moving
• Defocus blur NEXT WEEK
• Depth of field effects

4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
Overview

• Removing Camera Shake
• Non-blind
• Blind
• Removing Motion Blur
• Non-blind
• Blind
• Focus on software approaches

8
Lets take a photo
Blurry result
9
Slow-motion replay
10
Slow-motion replay
Motion of camera
11
Image formation model Convolution
?

Blur kernel
Blurry image
Sharp image
Input to algorithm
Desired output
Convolutionoperator
Model is approximation Assume static scene
12
Blind vs Non-blind

• Non-blind
• Blind

?
?
13
Camera Shake is it a convolution?
8 different people, handholding camera, using 1
second exposure
14
Dots from each corner
Person 1
Person 2
Topright
Topleft
Bot.right
Bot.left
Person 4
Person 3
15
What if scene not static?

• Partition the image into regions

16
Overview

• Removing Camera Shake
• Non-blind
• Blind
• Removing Motion Blur
• Non-blind
• Blind

17
Deconvolution is ill posed
?

Slide from Anat Levin
18
Deconvolution is ill posed
Solution 1

?
Solution 2

?
Slide from Anat Levin
19
Convolution- frequency domain representation
Sharp Image
1st observed image

Blur kernel
1-D Example
Output spectrum has zeros where filter spectrum
has zeros
Slide from Anat Levin
20
Idea 1 Natural images prior
What makes images special?
Natural
Unnatural
Image
gradient
Natural images have sparse gradients
put a penalty on gradients
Slide from Anat Levin
21
Deconvolution with prior
Convolution error
Derivatives prior
2
_

?
Low
Equal convolution error
2
_

?
High
22
Comparing deconvolution algorithms
(Non blind) deconvolution code available online
http//groups.csail.mit.edu/graphics/CodedAperture
/
Slide from Anat Levin
Input
23
Comparing deconvolution algorithms
(Non blind) deconvolution code available online
http//groups.csail.mit.edu/graphics/CodedAperture
/
Slide from Anat Levin
Input
spread gradients
localizes gradients
Richardson-Lucy
Gaussian prior
Sparse prior
24
(No Transcript)
25
(No Transcript)
26
Application Hubble Space Telescope

• Launched with flawed mirror
• Initially used deconvolution to correct images
  before corrective optics installed

Image of star
27
Non-Blind DeconvolutionMatlab Demo

• http//groups.csail.mit.edu/graphics/CodedAperture
  /DeconvolutionCode.html

28
Overview

• Removing Camera Shake
• Non-blind
• Blind
• Removing Motion Blur
• Non-blind
• Blind

29
Removing Camera Shake from a Single Photograph
Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T.
Roweis and William T. Freeman
Massachusetts Institute of Technology
andUniversity of Toronto
30
Overview
Joint work with B. Singh, A. Hertzmann, S.T.
Roweis W.T. Freeman
Original
Our algorithm
31
Close-up
Original
Naïve sharpening
Our algorithm
32
Image formation process
?

Blur kernel
Blurry image
Sharp image
Input to algorithm
Desired output
Convolutionoperator
Model is approximation Assume static scene
33
Existing work on image deblurring

• Old problem
• Trott, T., The Effect of Motion of
  Resolution,Photogrammetric Engineering, Vol.
  26, pp. 819-827, 1960.
• Slepian, D., Restoration of Photographs Blurred
  by Image Motion, Bell System Tech., Vol. 46, No.
  10, pp. 2353-2362, 1967.

34
Existing work on image deblurring

• Software algorithms for natural images

• Many require multiple images
• Mainly Fourier and/or Wavelet based
• Strong assumptions about blur ? not true for
  camera shake

• Image constraints are frequency-domain power-laws

35
Hardware approaches
Existing work on image deblurring
Coded shutter
Dual cameras
Image stabilizers
Raskar et al. SIGGRAPH 2006
Ben-Ezra Nayar CVPR 2004
Our approach can be combined with these hardware
methods
36
Why is this hard?
Simple analogy 11 is the product of two
numbers. What are they?
No unique solution 11 1 x 11 11 2 x
5.5 11 3 x 3.667 etc..
Need more information !!!!
37
Multiple possible solutions
Sharp image
Blur kernel

?
Blurry image
38
Natural image statistics
Characteristic distribution with heavy tails
Histogram of image gradients
Log pixels
39
Blurry images have different statistics
Histogram of image gradients
Log pixels
40
Parametric distribution
Histogram of image gradients
Log pixels
Use parametric model of sharp image statistics
41
Uses of natural image statistics

• Denoising Portilla et al. 2003, Roth and Black,
  CVPR 2005
• Superresolution Tappen et al., ICCV 2003
• Intrinsic images Weiss, ICCV 2001
• Inpainting Levin et al., ICCV 2003
• Reflections Levin and Weiss, ECCV 2004
• Video matting Apostoloff Fitzgibbon, CVPR
  2005
• Corruption process assumed known

42
Three sources of information

• 1. Reconstruction constraint

2. Image prior
Distribution of gradients
43
Three sources of information
y observed image b blur kernel x
sharp image
44
Three sources of information
y observed image b blur kernel x
sharp image
Posterior
45
Three sources of information
y observed image b blur kernel x
sharp image
Posterior
2. Image prior
1. Likelihood (Reconstruction constraint)
3. Blurprior
46
1. Likelihood
y observed image b blur x sharp image

• Reconstruction constraint

i - pixel index
47
2. Image prior
y observed image b blur x sharp image

• Mixture of Gaussians fit to empirical
  distribution of image gradients

Log pixels
i - pixel index c - mixture component index f
- derivative filter
48
3. Blur prior
y observed image b blur x sharp image

• Mixture of Exponentials
• Positive sparse
• No connectivity constraint

Most elements near zero
A few can be large
j - blur kernel element d - mixture component
index
49
The obvious thing to do
Posterior
2. Image prior
1. Likelihood (Reconstruction constraint)
3. Blurprior

• Combine 3 terms into an objective function
• Run conjugate gradient descent
• This is Maximum a-Posteriori (MAP)

No success!
50
Variational Bayesian approach

• Keeps track of uncertainty in estimates of image
  and blur by using a distribution instead of a
  single estimate

Optimization surface for a single variable
Maximum a-Posteriori (MAP)
Score
Variational Bayes
Pixel intensity
51
Variational Independent Component Analysis
Miskin and Mackay, 2000

• Binary images
• Priors on intensities
• Small, synthetic blurs
• Not applicable to natural images

52
Overview of algorithm
Input image

• Pre-processing
• Kernel estimation
• Multi-scale approach
• Image reconstruction
• - Standard non-blind deconvolution routine

53
Digital image formation process
Gammacorrection
RAW values
Remapped values
Blur process applied here
P. Debevec J. Malik, Recovering High Dynamic
Range Radiance Maps from Photographs, SIGGRAPH
97
54
Preprocessing
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image

• Bayesian inference too slow to run on whole
  image

Infer kernel from this patch
55
Initialization
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image
Initialize 3x3 blur kernel
Initial image estimate
Initial blur kernel
Blurry patch
56
Inferring the kernel multiscale method
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image
Loop over scales
VariationalBayes
Upsampleestimates
Initialize 3x3 blur kernel
Use multi-scale approach to avoid local minima
57
Image Reconstruction
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image
Loop over scales
VariationalBayes
Upsampleestimates
Initialize 3x3 blur kernel
Full resolutionblur estimate
Non-blind deconvolution (Richardson-Lucy)
Deblurred image
58
Syntheticexperiments
59
Synthetic example
Artificial blur trajectory
Sharp image
60
Synthetic blurry image
61
Inference initial scale
Image before
Image after
Kernel before
Kernel after
62
Inference scale 2
Image before
Image after
Kernel after
Kernel before
63
Inference scale 3
Image before
Image after
Kernel after
Kernel before
64
Inference scale 4
Image before
Image after
Kernel before
Kernel after
65
Inference scale 5
Image before
Image after
Kernel after
Kernel before
66
Inference scale 6
Image before
Image after
Kernel after
Kernel before
67
Inference Final scale
Image before
Image after
Kernel after
Kernel before
68
Comparison of kernels
True kernel
Estimated kernel
69
Blurry image
70
Matlabs deconvblind
71
Blurry image
72
Our output
73
True sharp image
74
What we do and dont model

• DO
• Gamma correction
• Tone response curve (if known)
• DONT
• Saturation
• Jpeg artifacts
• Scene motion
• Color channel correlations

75
Realexperiments
76
Results on real images

• Submitted by people from their own photo
  collections
• Type of camera unknown
• Output does contain artifacts
• Increased noise
• Ringing
• Compare with existing methods

77
(No Transcript)
78
(No Transcript)
79
Close-up

• Original
• Output

80
Original photograph
81
Blur kernel
Our output
82
Matlabs deconvblind
83
Close-up
Matlabs deconvblind
Original
Our output
84
Original photograph
85
Our output
Blur kernel
86
Photoshop sharpen more
87
Original image
Close-up
Close-up of our output
Blur kernel
Close-up of image
88
Original photograph
89
Our output
Blur kernel
90
Original image
91
Our output
Blur kernel
92
Close-up
Blur kernel
Our output
Original image
93
What about a sharp image?
Original photograph
94
Our output
Blur kernel
95
Original photograph
96
Blur kernel
Our output
97
Close-up
Blur kernel
Our output
Original image
98
Original photograph
99
Blurry image patch
Our output
Blur kernel
100
Original photograph
101
Our output
Blur kernel
102
Close-up of bird
Original
Unsharp mask
Our output
103
Original photograph
104
Blur kernel
Our output
105
Image artifacts estimated kernels
Blur kernels
Image patterns
Note blur kernels were inferred from large image
patches, NOT the image patterns shown
106
Code available online

• http//cs.nyu.edu/fergus/research/deblur.html

107
Summary

• Method for removing camera shake from real
  photographs
• First method that can handle complicated blur
  kernels
• Uses natural image statistics
• Non-blind deconvolution currently simplistic
• Things we have yet to model
• Correlations in colors, scales, kernel continuity
• JPEG noise, saturation, object motion

108
Overview

• Removing Camera Shake
• Non-blind
• Blind
• Removing Motion Blur
• Non-blind
• Blind

109
(No Transcript)
110
(No Transcript)
111
(No Transcript)
112
Deblurred Result
Input Photo
113
Traditional Camera Shutter is OPEN
114
Our Camera Flutter Shutter
115
Shutter is OPEN and CLOSED
116
Comparison of Blurred Images
117
Implementation Completely Portable
118
Lab Setup
119
Blurring Convolution
Sync Function
Traditional Camera Box Filter
120
Preserves High Frequencies!!!
Flutter Shutter Coded Filter
121
Comparison
122
Inverse Filter stable
Inverse Filter Unstable
123
Short Exposure
Long Exposure
Coded Exposure
Our result
Ground Truth
Matlab Lucy
124
Overview

• Removing Camera Shake
• Non-blind
• Blind
• Removing Motion Blur
• Non-blind
• Blind

125
(No Transcript)
126
Use statistics to determine blur size

• Assumes direction of blur known

127
Input image
128
Deblur whole image at once
129
Local Evidence
130
Proposed boundary
131
Result image
132
Input image (for comparison)
133
(No Transcript)
134
Let y 2 s2 0.1
135
Gaussian distribution
136
(No Transcript)
137
Marginal distribution p(by)
138
MAP solution

• Highest point on surface

139
MAP solution

• Highest point on surface

140
Variational Bayes

• True Bayesian approach not tractable
• Approximate posterior with simple distribution

141
Fitting posterior with a Gaussian

• Approximating distribution is
  Gaussian
• Minimize

142
KL-Distance vs Gaussian width
143
Fitting posterior with a Gaussian

• Approximating distribution is
  Gaussian
• Minimize

144
Variational Approximation of Marginal
Variational
True marginal
MAP
145
Try sampling from the model

• Let true b 2
• Repeat
• Sample x N(0,2)
• Sample n N(0,s2)
• y xb n
• Compute pMAP(by), pBayes(by)
  pVariational(by)
• Multiply with existing density estimates (assume
  iid)

146
Setup of Variational Approach
Work in gradient domain

• Approximate posterior
  with

Assume
is Gaussian on each pixel
is rectified Gaussian on each blur kernel element
Cost function