Video Superresolution

1
Video Superresolution

• 18-798
• Anup Doshi and Jae K. Lee

2
Algorithm

• Our super-resolution algorithm is based mostly
  upon algorithm described in 1 for pure
  translational motion and space-invariant blur.
• General Idea Given a sequence of low-resolution
  images, we can model them as a set of linear
  operators on a single high-resolution image.
• Thus it is possible to reconstruct the
  high-resolution image by applying the inverse
  operators
• 1 Elad, M. Hel-Or, Y. A fast super-resolution
  reconstruction algorithm for pure translational
  motion and common space-invariant blur Image
  Processing, IEEE Transactions on, Vol.10, Iss.8,
  Aug 2001Pages1187-1193

3
Algorithm from paper

• Estimate translational motion of low-res images
  with respect to base frame
• Model motion, blur, and decimation as linear
  transforms
• Lexicographically order blocks and apply
  necessary inverse transforms
• De-blur if necessary (assuming blur is
  space-invariant)
• Apply iteratively to enhance video sequence

4
Algorithm Specifics

• Define Hi-res image (vector) X and Low-res images
  (vectors) Yk, k1..N.
• Define Linear Matrix Operations
• Decimation Dk, Geometric Warp Fk, Blur Hk
• Then we assume the model
• Yk DkHkFkX
• Further Assumptions Hk, Dk constant for all k
  warp is pure translation
• Yk(pqx1) D(pqxmn)H(mnxmn)F(mnxmn)X(mnx1)

5
Algorithm Specifics

• Conclusions (from 1)
• We can apply inverse operations to construct X
  given Fks, D, H, and Yks
• Let
• Then X H-1(R-1P).
• De-blurring can be applied separately

6
Our Work Expanding on Prior Results

• Problem Assumption of pure translational motion
  is very limiting
• Solution Use block-based algorithm
• Split each frame into smaller blocks operate on
  each block
• Also eases computational load
• Allows for slightly different motions within
  frames.

7
Our Work Expanding on Prior Results

• Problems with Motion Estimation (discussed below)
• Problems with Block-based algorithm Takes long
  time in Matlab. (discussed below)

8
Optical Flow Motion Estimation

• Precise, but not accurate
• Took too long for us
• Decided to go with Phase Correlation
• Less precise, more consistent
• Our algorithm does not work well for subpixel
  motion anyway

9
(No Transcript)
10
(No Transcript)
11
Results

• Algorithm took a lot longer than expected to
  process and compute
• Had to use smaller blocks for block-based
  processing
• This had an effect on the maximum motion we could
  handle between frames

12
Results (cont)

• Stuck with upsampling by four to keep problem
  simpler.
• The frames in the flower sequence tended to have
  a lot of motion between them
• This caused problems for the motion estimator and
  block-based methods.
• See output testout_all

13
Results (cont)

• Decided to limit the maximum motion to a range of
  -5,5 in x and y directions
• Got mostly decent results
• Subtle problems occur when there are no close
  frames found the frame then chosen may overlap
  with the base and then the Y-channel intensity
  will be double what it should be, causing an
  excess of pink to show up.
• Problem mostly taken care of, but sometimes still
  occursextremely subtle issue and tough to
  program around gracefully
• See results posttest_all, posttest_all4

14
Results (cont)

• Continuous updates results were really bad if we
  tried using previous Superres frame for next one,
  since errors in estimation compounded
• Blurring we found that it took a pretty long
  time to deblur, and most frames had more errors
  due to subpixel motion/motion estimation errors.