Title: Dynamic vision copes with
1Dynamic Vision
- Dynamic vision copes with
- Moving or changing objects (size, structure,
shape) - Changing illumination
- Changing viewpoints
- Input sequence of image frames
- Frame Image at a particular instant of time
- Differences between frames due to motion of
camera or object, illumination changes, changes
of objects - Output detect changes, compute motion of camera
or object, recognize moving objects etc.
2- There are four possibilities
- Stationary Camera, Stationary Objects (SCSO)
- Stationary Camera, Moving Objects (SCMO)
- Moving Camera, Stationary Objects (MCSO)
- Moving Camera, Moving Objects (MCMO)
- Different techniques for each case
- SCSO is simply static scene analysis simplest
- MCSO most general and complex case
- MCSO, MCMO in navigation applications
- Dynamic scene analysis ? more info ? can be
easier than static scene analysis
3- Frame sequence F(x,y,t)
- Intensity of pixel x, y at time t
- Assume that t represents the t-th frame
- The image is acquired by camera at the origin of
the 3-D coordinate system - Detect changes in F(x,y,t) between successive
frames - At pixel, edge, region level
- Aggregate changes to obtain useful info (e.g.,
trajectories)
4- Difference Pictures Compare the pixels of two
frames - Where t is a user defined threshold
- Pixels with value 1 result from motion or
illumination changes - Assumes that the frames are properly registered
- Thresholding is important slow moving objects
may not be detected for a given t
5(a), (b) frames from a sequence with moved
objects (c ) their difference thresholded with
t25
(a), (b) frames from a sequence with change in
illumination (c ) their difference thresholded
with t25
6- Size filtering only pixels that belong to a 4 or
8 connected component with intensity larger than
t are retained - Result of size filtering with t 10
- Removes mainly noisy regions
7- Robust change detection intensity
characteristics of regions are compared using a
statistical criterion - Super-pixels nxm non-overlapping rectangles
- Local mask groups of pixels of local pixel area
8- Super-pixels
- Mask of local pixel area
- Robust change detection with
- Super-pixels (super-pixel
- Resolution)
- Pixel masks
9- Accumulative difference pictures analyze changes
over a sequence of frames - Compare every frame with a reference frame
- Increase a difference term by 1 whenever the
difference is greater than the threshold - Detects even small or slowly moving objects
- Eliminates small misregistration between frames
10Change detection using accumulative
differences (a),(b) first and last frames (c)
accumulative difference picture
11- Segmentation using motion find objects in SCMO
and MCMO scenes - SCMO separate moving objects from stationary
background - MCMO remove the motion due to camera
- Correspondence problem the process of
identifying the same object of feature in two or
more frames - Large number of features ? put restrictions on
the number of possible matches - Regions, corners, edges
12- Temporal and Spatial Gradients Compute
- dF/ds spatial gradient
- dF/dt temporal gradient
- Apply threshold to the product
- Responds even to slow moving edges
13(a),(b) two frames of a sequence (c) edges
detected using spatial-temporal gradients
14- Using difference pictures (stationary camera)
difference and accumulative difference pictures
find moving areas
15- The area in PADP, NADP is the area covered by the
moving object in the reference frame - PADP, NADP continue to increase in value but the
regions stops growing in size - Use a mask of the object to determine whether a
region is growing - Masks can be obtained from AADP when the object
has been completely displaced - In cases of occlusion monitor changes in regions
16(a)-(c) frames 1,5,7, containing a moving
object (d),(e),(f) PADP, NADP, AADP
17- Motion correspondence to determine the motion of
objects establish a correspondence between
features in two frames - Correspondence problem pair a point pi(xi,yi)
in the first image with a point pj(xj,yj) in the
second image - Disparity dij(xi - xj,yi - yj)
- Compute disparity using relaxation labeling
- Questions how are points selected for matching?
how are the correct matches chosen? what
constraints?
18- Three properties guide matching
- Discreteness minimize expensive searching
(detect points at which intensity values are
varying quickly in at least one direction) - Similarity match similar features (e.g.,
corners, edges, etc.) - Consistency match nearby points (a point cannot
move everywhere)
correspondence problem as bipartite graph
matching between two frames A, B remove all but
one connection for each point
19- Disparity computation using Relaxation Labeling
- Identify the features to be matched
- E.g., corners or (generally) points i, j
- Let Pij0 be the initial probability
- Disparity dij(xi - xj,yi - yj) lt Dmax
high probabilities for points in dx with similar
motion
points cannot move everywhere
neighborhood
20- Update Pij at every iteration
- For every point i, the algorithm computes
- i, (dxij,Pij)0, (dxij,Pij)1,. (dxij,Pij)n
- n n-th iteration or frame
- Use correspondences with high Pij
- Use these for the next two frames etc.
210 1 3 4
n
From Ballard and Brown, 94
Disparities after 3 iterations
22disparities after Applying a relaxation labeling
algorithm
23- Image flow velocity field in the image due to
motion of the observer, objects or both - Velocity vector of each pixel
- Image flow is computed at each pixel
- SCMO most pixels will have zero velocity
- Methods pixel and feature based methods compute
image flow for all pixels or for specific
features (e.g., corners) - Relaxation labeling methods
- Gradient Based methods
24- Gradient Based Methods exploit the relationship
between spatial and temporal gradients of
intensity - Assumption continuous and smooth changes of
intensity in consecutive frames
Taylor expansion
25- Better estimation the error term is not zero
- It has to be minimum Apply Lagrange multipliers
method - fx, fy,ft derivatives of F with regard to x,y
and t - The derivative of F2 with regard to x,y is zero
- From Ballard and Brown 1984 ?
- The computation can be unreliable at motion
boundaries (e.g., occluded boundaries)
26Turn this into an iterative method for solving
ux,uy?
27- Optical flow computation for two consecutive
frames (Horn Schunck 1980) - k0
- Initialize all uxkuyk0
- Repeat until some error criterion in satisfied
28- Multi-frame optical flow compute optical flow
for two frames and use this to initialize optical
flow for the next frame etc. - k0
- Initialize all ux,uy (apply the previous
algorithm) - Repeat until some error criterion in satisfied
29from Ballard and Brown84
(a),(b),(c) three frames from a rotating
sphere (d) optical flow after 32 frames
30- Information in optical flow assuming high
quality computation of optical flow - Areas of smooth velocity ? single surfaces
- Areas with large gradients ? occlusion and
boundaries - The translation component of motion is directed
toward the Focus Of Expansion (FOE) intersection
of the directions of optical flow as seen by a
moving observer - Surface structure can be derived from the
derivatives of the translation component - Angular velocity can be determined from the
rotational component
31- The velocity vectors of the stationary components
of a scene as seen by a translating observer meet
at the Focus Of Expansion (FOE)
32- Tracking a feature or an object must be tracked
over a sequence of frames - Easy to solve for single entities
- For many entities moving independently requires
constraints - Path coherence the motion of an object in a
frame sequence doesn't change abruptly between
frames (assumes high sampling rate) - The location of a point will be relatively
unchanged - The scalar velocity of a point will be relatively
unchanged - The direction of motion will be relatively
unchanged
33- Deviation function for path coherence
- A trajectory of a point i TiltPi1,Pi2,,Pingt
- Pik point i at k-th frame
- In vector form XiltXi1,Xi2,,Xingt
- Deviation of the point in the k-th frame
dikf(Xik-1Xik,XikXik1) - Deviation for the complete trajectory
- For m points (trajectories) in a sequence of n
frames the total deviation is - Minimize D to find the correct trajectories
34- The trajectories of 2 points The points in the
1st, 2nd - and 3rd frame are labeled by squares, triangles
and - rhombus respectively
- The change in the direction and velocity must be
smooth
35- D is a function of f, how f is computed?
- It is described by the function
- f can also be written as
w1,w2 weight terms
36- Direction coherence the first term
- Dot product of displacement vectors
- Speed coherence the second term
- Geometric/arithmetic mean of magnitude
- Limitations same number of features in every
frame - Objects may disappear, appear or occlude
- Changes of geometry, illumination
- Lead to false correspondences
- Force the trajectories to satisfy certain local
constraints