Image Mosaicing with Motion Segmentation from Video - PowerPoint PPT Presentation

About This Presentation
Title:

Image Mosaicing with Motion Segmentation from Video

Description:

To create a single panoramic image from a sequence of video frames ... Need a motion vector for every pixel. ... The Whole Enchilada ... – PowerPoint PPT presentation

Number of Views:73
Avg rating:3.0/5.0
Slides: 24
Provided by: talyg
Learn more at: https://web.stanford.edu
Category:

less

Transcript and Presenter's Notes

Title: Image Mosaicing with Motion Segmentation from Video


1
Image Mosaicing with Motion Segmentation from
Video
  • Augusto Roman, Taly Gilat
  • EE392J Final Project
  • 03/20/01

2
Project Goal
  • To create a single panoramic image from a
    sequence of video frames
  • Align frames using motion estimation

3
The Wang Adelson Algorithm
  • Dense MV is gold standard estimate for each pixel
  • Estimate region motion
  • Segment regions based on motions
  • Iterate

4
Dense Motion Estimation
  • Need a motion vector for every pixel.
  • Full search block matching is far too slow (more
    than 2 hours per frame pair!)
  • Phase Correlation used to estimate motion
  • Results in a few possible motions for each block.
  • For each pixel in the center of the block, test
    with each of the possible motions and choose the
    motion with the smallest MSE.
  • Shift block by small amount and compute new phase
    correlation and repeat.
  • Much faster! 30 sec per frame pair!

5
Phase Correlation
  • Given a block B1,B2 from each image
  • Compute 2D FFT of each
  • Compute cross-power spectrum
  • Normalized value of FB1 FB2
  • Take IFFT to get Phase Correlation Function
  • This is very similar to the correlation function
    between the two blocks

6
Motion Vectors vs Motions Hypothesis
  • There are a number of different motion hypotheses
    available.
  • In theory, each of these hypothesis corresponds
    to a distinct motion in the video.
  • Each pixel is assigned to the motion hypothesis
    that most closely approximates its motion vector.
  • This segments the frame into distinct regions,
    one for each motion hypothesis.

7
Motion Hypothesis Generation
  • For each region want motion hypothesis that best
    represents all pixel motions in that region.
  • Least squares fit to find best affine motion
    parameter in a region
  • First iteration initialized with blocks

8
Motion Hypothesis Refinement
  • K-means used to cluster motion hypotheses
  • K unknown
  • Empty clusters removed
  • Large clusters split to maintain minimum k value

9
Region Segmentation
  • For each pixel compare hypotheses to dense MV
  • Find closest hypothesis
  • Group all pixels represented by a motion
    hypothesis into a region
  • Pixels with large error unassigned
  • Hypotheses without membership removed

10
Region Adjustment
  • Region Splitter
  • Assumes areas with same motion are connected
  • Disconnected areas within a region are split
    into separate regions
  • Increases number of hypotheses for k-means
  • Region Filter
  • Small regions give poor motion estimates
  • Remove all regions with area below threshold
  • Disconnected objects with same motion will be
    merged at next segmentation step

11
Segmentation Results
Frame 1
Frame 2
12
Segmentation Results
Initial Segmentation
Iteration 1
Iteration 2
Iteration 3
Iteration 5
Iteration 7
13
The Whole Enchilada
  • The dense motion estimate, region segmentation
    and motion estimation performed for all frames
    pairs.
  • For first pair, segmentation initialized to
    blocks and k-means initialized to lattice in 6D
    affine space.
  • Subsequent frame pairs initialized with final
    segmentation and motion hypotheses from previous
    frame pair.

14
Layer Synthesis
  • Motion estimates relate each frame only to the
    previous frame
  • Frames are projected onto first video frame
  • Cumulative projection kept in 3x3 transformation
    matrix
  • Layers are not necessarily ordered similarly
    between frames
  • Assumed largest layer is background
  • Median taken of all values projected to each
    pixel in final image

15
Implementation Difficulties
  • Parameter tweaking
  • Phase Correlation
  • Attempted to focus on textured areas of image
  • Initial test videos lacked texture
  • Registering layers across frames
  • Cumulative motion error
  • Resolution loss in final panoramic

16
Results
17
Results
18
Results
19
Results
20
Results
21
Results
22
Results
23
Conclusions
  • Nice panoramas
  • Moving objects removed
  • Future improvements
  • Sub-pixel dense motion estimation
  • More accurate segmentation
  • Region registration across frame
Write a Comment
User Comments (0)
About PowerShow.com