Image Enhancement by Regularization Methods

1
Image Enhancement by Regularization Methods
Andrey S. Krylov, Andrey V. Nasonov, Alexey S.
Lukin
Moscow State University Faculty of Computational
Mathematics and Cybernetics Laboratory of
Mathematical Methods of Image Processing
2
Introduction

• Many image processing problems are posed as
  ill-posed inverse problems. To solve these
  problems numerically one must introduce some
  additional information about the solution, such
  as an assumption on the smoothness or a bound on
  the norm.

This process was theoretically proven by Russian
mathematician Andrey N. Tikhonov and it is
known as regularization.
3
Outline

• Regularization methods
• Applications
• Resampling (interpolation)
• Deringing (Gibbs effect reduction)
• Super-resolution

4
Ill-posed Problems

• Formally, a problem of mathematical physics is
  called well-posed or well-posed in the sense of
  Hadamard if it fulfills the following conditions
• 1. For all admissible data, a solution exists.
• 2. For all admissible data, the solution is
  unique.
• 3. The solution depends continuously on data.

5
Ill-posed Problems

• Many problems can be posed as problems of
  solution of an equation
• A is a linear continuous operator, Z and U are
  Hilbert spaces
• The problem is ill-posed and the corresponding
  matrix for operator ? in discrete form is
  ill-conditioned

6
Point Spread Function (PSF)
Assume Point light source
7
Convolution Model

• Notations
• L original image
• K the blur kernel (PSF)
• N sensor noise (white)
• B input blurred image

?

8
Deblur using Convolution Theorem
Convolution Theorem
9
Deblur using Convolution Theorem
Blurred Image
PSF
10
Noisy case
11
Variational regularization methods

• Tikhonov methods
• The Residual method (Philips)
• The Quasi-solution method (Ivanov)

12
Variational regularization methods

• Regularization method is determined by
• A) Choice of solution space and of stabilizer
• B) Choice of
• C) Method of minimization
• A and B determine additional information on
  problem solution we want to use for solution of
  ill-posed problem to achieve stability

13
Outline

• Regularization methods
• Applications
• Resampling (interpolation)
• Deringing (Gibbs effect reduction)
• Super-resolution

14
ResamplingIntroduction

• Interpolation is also referred to as resampling,
  resizing or scaling of digital images
• Methods
• Linear non-adaptive (bilinear, bicubic, Lanczos
  interpolation)
• Non-linear edge-adaptive (triangulation, gradient
  methods, NEDI)
• Regularization method is used to construct a
  non-linear edge-adaptive algorithm

15
ResamplingLinear and non-linear method

• bilinear interpolation

• non-linear method

16
ResamplingInverse problem

• Consider the problem of resampling as
• Problem operator A is not invertible

z is unknown high-resolution image, u is
known low-resolution image, A is the
downsampling operator which consists of filtering
H and decimation D
17
ResamplingRegularization

• We use Tikhonov-based regularization
  methodwhere

18
ResamplingRegularization

• Choices of regularizing term (stabilizer)
• Total Variation
• Bilateral TV and are shift operators
  along x and y axes by s and t pixels
  respectively, p 1, ? 0.8

19
ResamplingRegularization

• Minimization problem
• Subgradient method

20
ResamplingResults

• Linear method

• Regularization-basedmethod

• Gibbs phenomenon

21
Image Enhancement by Regularization Methods

• Introduction to regularization
• Applications
• Resampling (interpolation)
• Deringing (Gibbs effect reduction)
• Super-resolution

22
Total Variation Approach for Deringing

• Gibbs effect is related to Total Variation

High TV, very notable Gibbs effect (ringing)
Low TV
23
Total Variation Regularization methods

• Tikhonovs approach
• Rudin, Osher, Fatemi method
• Ivanovs quasi-solution method

24
Global and Local Deringing

• Two approaches for Deringing
• Global deringing
• Minimizes TV for entire image
• In this case, we use Tikhonov regularization
  method
• No ways to estimate regularization parameter,
  details outside edges may be lost
• Local deringing
• Used if we have information on TV for small
  rectangular areas
• In this case, we use Ivanovs quasi-solution
  method for small overlapping blocks

25
Deringing after interpolation

• Deringing after interpolation
• We know information on TV for blocks of initial
  image to be resampled
• We suggest that TV does not change after image
  interpolation

Thus we have real algorithm to find
regularization parameter for deringing after
image resampling task
26
Minimization

• Tikhonov regularization method
• Subgradient method
• Quasi-solution method
• 1D Conditional gradient method (there is no
  effective 2D implementation)
• In 2D case, we divide an image into a set of rows
  and process these rows by 1D method, next we do
  the same with columns and finally we average
  these results

27
Minimization

• Conditional gradient method
• Conditional gradient method is used to minimize a
  convex functional on a convex compact set. The
  key idea of this method is that step directions
  are chosen among the vertices of the set of
  constraints, so we do not fall outside this set
  during minimization process
• Conditional gradient method is effective only for
  small images, so it is used for local deringing
  only

28
Resampling DeringingPSNR Results
A set of 100 nature and architecture images
with 400x300 resolution (11x11 blocks, 1813 per
image) was used to test the methods. We
downsampled the images by 2x2 using Gauss blur
with radius 0.7 and then upsampled them by our
regularization algorithm. Next we applied
deringing methods and compared the results with
initial images.
29
Resampling DeringingResults

• regularization-basedinterpolation

• application of quasi-solution method

30
Resampling DeringingResults
Regularization-based interpolation
quasi-solution deringing method
Source image, upsampled by box filter
Linear interpolation
Regularization-based method
31
Image Enhancement by Regularization Methods

• Introduction to regularization
• Applications
• Resampling (interpolation)
• Deringing (Gibbs effect reduction)
• Super-resolution

32
Super-ResolutionIntroduction

• The problem of super-resolution is to recover a
  high-resolution image from a set of several
  degraded low-resolution images
• Super-resolution methods
• Learning-based single image super-resolution,
  learning database (matching between low- and
  high-resolution images)
• Reconstruction-based use only a set of
  low-resolution images to construct
  high-resolution image

33
Super-ResolutionInverse Problem

• The problem of super-resolution is posed as error
  minimization problem
• z reconstructed high-resolution image
• vk k-th low-resolution input image
• Ak downsampling operator, it includes motion
  information

34
Super-ResolutionDownsampling operator

• Ak downsampling operator
• Hcam camera lens blur (modeled by Gauss filter)
• Hatm atmosphere turbulence effect (neglected)
• n noise (ignored)
• Fk warping operator motion deformation

35
Super-ResolutionWarping operator

• Warping operator Fk

36
Super-ResolutionRegularization

• The problem is ill-posed, and we use Tikhonov
  regularization approach (same as in
  resampling)
• where ,
• Minimization by subgradient method

37
Super-ResolutionResults
Source images
Linear method
Pixel replication
Non-linear method
Super-resolution
Face super-resolution for the factor of 4 and 10
input images
38
Super-ResolutionResults
linearly interpolated single frame
examples of input frames (of total 14)
super-resolution result

• The reconstruction of an image from a sequence

39
Super-Resolution for Video
current frame
For every frame, we take current frame, 3
previous and 3 next frames. Then we process it by
super-resolution.
40
Super-ResolutionResults for Video
Nearest neighbor interpolation
Super-Resolution
Super-Resolution for video for a factor of 4
41
Super-ResolutionResults for Video
Bilinear interpolation
Super-Resolution
Super-Resolution for video for a factor of 4
42
Super-ResolutionResults for Video
Bicubic interpolation
Super-Resolution
Super-Resolution for video for a factor of 4
43
Conclusion

• Increasing CPU and GPU power makes regularization
  methods more and more important in image
  processing
• Regularization is a very powerful tool but each
  specific image processing problem needs its own
  regularization method

44
Thank you!

• http//imaging.cs.msu.ru/