SEGMORA : Mobile Segmentation for Augmented Reality

1
SEGMORA Mobile Segmentation for Augmented
Reality
Cédric De Roover, Moncef Gabbouj, Benoît Macq

• Université catholique de Louvain
• Tampereen Teknillinen Yliopisto

2
Project goal

3
Augmented reality with moving camera

• 1 The camera path is evaluted

• 2 The 3D oject is created and then the camera
  path is applied on it

• 3 A mask is performed

• 4 Special effects are added to the movie scene
  

4
Augmented reality with moving camera

• For inserting objects in augmented reality behind
  people, we need a mask to select people at the
  foreground.
• Nowadays, two solutions
• Do it by hand, frame by frame
• Film people before a blue or green screen

5
Goal of the project

• The goal of the SEGMORA project is to develop an
  algorithm of segmentation to create a mask in
  video sequences with moving camera.

6
A general framework

7
A general framework

• Image segmentation
• - Based on region (watershed, region growing)
• - Based on active contours (level sets)
• 2D1 segmentation
• - Spatial segmentation 2D
• - Temporal segmentation 1D
• Constrains
• - accuracy and no real time
• - interactions with the user is allowed

8
A general framework

9
Global motion compensation and local motion
estimation
10
Global Motion Compensation

• Goal compensate the global motion and estimate
  the local motion after the compensation
• Motion estimation between t and t-1
• Compensation of t-1 thanks to the global motion
  equation
• Motion estimation between t et t-1compensated

11
Global Motion Compensation

• Motion estimation (optical flow)
• Input Image t and Image t-1
• Output Motion vectors between t and t-1
• Example Block Matching Algorithm (full search,
  three steps)

12
Global Motion Compensation

• Improvement by filtering
• Suppress vectors that are too much different from
  their neighbours
• Do not compensate the global motion if the
  compensation is too small
• Perform the parameters estimation only on
  background

13
Spatial Segmentation
14
Spatial segmentation

• Pre-filtering
• Input original frame
• Output filtered frame
• Example Opening - Closing
• Compute the gradient image
• Input filtered image
• Output gradient image
• Examples - Morphological gradient
• - 2D first derivative

15
Spatial segmentation

• Watershed
• Input Gradient image
• Output A label for every pixel
• Example Vincent and Soille Algorithm
• gt over-segmentation (here 285 regions)

16
Spatial segmentation

• Region Adjacency Graph (RAG)
• Input Original image labels
• Output Graph representation
• Informations
• Region size
• Mean color
• Mean motion
• Neighbour regions
• Number of pixels on the boundary
• gt informations are needed by the fusion part

17
Temporal Segmentation
18
Temporal segmentation

• Find the motion areas
• By using the local motion estimation
• ) Motion direction
• -) Not dense field
• By using the Change Detection Mask
• - If no motion I(t)-I(t-1)d(t) is Gaussian
• - We compare a local sum of normalized
  differences
• gt chi square distribution with N liberty degree
  gt threshold determination
• N25 gt 37,65 (0,05) 44,31 (0,01)

19
Fusion and binary tree
20
Fusion and binary tree

• We need to determine the order to fuse the
  regions
• Fusion based on the regions size
• Fusion based on the region volume during the
  watershed
• Other possibility, binary tree.

21
Fusion and binary tree

• Binary tree creation
• Input Graph original images t and t-1
  motion estimation
• Output A region binary tree
• Two adjacent nodes are the two nodes which are
  the most similars ...
• Example Similarity Measure
• SM alpha SMSpatial beta SMTemporal
• SMS Mean intensity difference between the 2
  regions at time t
• SMT Mean differences difference between 2
  regions at time t and t-1

22
Tracking and Mask computation
23
Tracking and mask computation

• Mask computation
• Input Graph (after fusion) originals frames
  originales
• Output Binary mask
• Example Markov Random Field
• minimize the energy function
• with

24
Interactions
25
Interactions

• Possibility for the user to
• - specify the background and foreground
• - change the label of regions and split the too
  big regions

26
Results
27
References

• M. Kim, J.G. Choi, D. Kim, H. Lee, M.H. Lee, C.
  Ahn, and Y. Ho A VOP Generation Tool
  Automatic Segmentation of Moving Objects in Image
  Sequences Based on Spatio-Temporal Information,
  IEEE Trans. on CSVT, vol. 9, No. 8, December
  1999.
• Y. Tsaig and A. Averbuch Automatic
  Segmentation of Moving Objects in Video
  Sequences A Region Labeling Approach, IEEE
  Trans. on CSVT, vol. 12, No. 7, July 2002.
• L. Vincent and P. Soille Watersheds in Digital
  Spaces An Efficient Algorithm Based on Immersion
  Simulations, IEEE Trans. on Patern Analysis and
  Machine Intelligence, vol. 13, No. 6, June 1991.

28
Partners of the project

• Catholic University of Louvain, Louvain-la-Neuve,
  Belgium (Prof. Benoît Macq, Cédric De Roover)
• Tampere University of Technology, Finland (Prof.
  Moncef Gabbouj)
• Axell Communication, Resteigne, Belgium (Philippe
  Axell)
• Alterface, Louvain-la-Neuve, Belgium (Dr. Xavier
  Marichal)