Using Photographs to Enhance Videos of a Static Scene

1
Using Photographs to Enhance Videos of a Static
Scene

• Pravin Bhat1, C. Lawrence Zitnick2, Noah
  Snavely1, Aseem Agarwala3, Maneesh Agrawala4,
  Michael Cohen1,2, Brian Curless1, Sing Bing
  Kang2

University of Washington1, Microsoft Research
Redmond2 University of California3, Adobe
Systems4
EGSR 2007
2
An overview of Spacetime Fusion
3
Motivation

• Low quality video

Input Video
4
Motivation

• Low quality video
• Reconstructed video

Input Video
Reconstructed Video
5
Motivation

• Low quality video
• Reconstructed video
• Reconstructed from photos
• Good spatial reconstruction
• Bad temporal reconstruction

Input Video
Reconstructed Video
6
Motivation

• Spacetime Fusion result

Input Video
Spacetime Fusion Result
7
Motivation

• Spacetime Fusion result
• Spatial properties of reconstruction
• Temporal properties of input video

Input Video
Spacetime Fusion Result
8
Spacetime Fusion

• Define a 3D gradient field

9
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video

10
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video
• Key Idea
• Temporal gradients defined betweenmotion
  compensated temporal neighbors

11
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video
• Key Idea
• Temporal gradients defined betweenmotion
  compensated temporal neighbors

Video frame t - 1
Video frame t
12
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video
• Key Idea
• Temporal gradients defined betweenmotion
  compensated temporal neighbors

Gt
Video frame t - 1
Video frame t
Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)
13
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video
• Key Idea
• Temporal gradients defined betweenmotion
  compensated temporal neighbors

Gt
Video frame t - 1
Video frame t
Gt(x, y, t) V(x, y, t) - V(x - u, y - v, t - 1)
14
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video
• Key Idea
• Temporal gradients defined betweenmotion
  compensated temporal neighbors
• Increases compatibility betweentemporal
  gradients and spatial gradients

15
Spacetime Fusion

• Define a 3D gradient field
• Spatial gradients from reconstruction
• Temporal gradients from input video
• Key Idea
• Temporal gradients defined betweenmotion
  compensated temporal neighbors
• Increases compatibility betweentemporal
  gradients and spatial gradients
• Integrate the 3D gradient field

16
Spacetime Fusion

• Integrating the gradient field

Solve linear system Av b
17
Spacetime Fusion

• Integrating the gradient field

Solve linear system Av b Constraints vx, y,
t vx-1, y, t Gx(x, y, t) vx, y, t vx,
y-1, t Gy(x, y, t) vx, y, t vx-u, y-v, t
Gt(x, y, t)
18
Applications
19
Enhanced Exposure
20
Edit Propagation
Input Video
21
Edit Propagation
User Edits
22
Edit Propagation
User Edits
23
Edit Propagation
User Edits
24
Edit Propagation
User Edits
25
Edit Propagation
User Edits
26
Edit Propagation
User Edits
27
Edit Propagation
Edited Video
28
Super-Resolution
29
Conclusion

• Spacetime fusion

30
Conclusion

• Spacetime fusion
• Combines spatial and temporal gradients from two
  different sources

31
Conclusion

• Spacetime fusion
• Combines spatial and temporal gradients from two
  different sources
• Requires motion vectors for temporal source
• stereo (static scenes)
• flow (dynamic scenes)

32
Conclusion

• Spacetime fusion
• Combines spatial and temporal gradients from two
  different sources
• Requires motion vectors for temporal source
• stereo (static scenes)
• flow (dynamic scenes)
• Major applications
• Enforcing temporal coherence
• Transferring lighting information