A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS

1
A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS

• Ginés García Mateos
• ginesgm_at_um.es
• Sergio Fructuoso Muñoz
• sergiofr_at_ono.com
• Dept. de Informática y Sistemas
• University of Murcia - SPAIN

2
System overview
Camera
Face detector and tracker
User of the per-ceptual interface
MOVEFORWARD/BACKWARDLEFT/RIGHT ROTATE LEFT/RIGHT
LOOKUP/DOWN
FACE LOCATIONX 6 Y 0 Z 3 FACE
ORIENTATION Roll 7.7 Pitch 0 Yaw 3 FACE
MOVEMENT?X -2 ?Y 1 ?Z 0
3D movement and pose estimation
Control signals (virtual world movement)
Virtual world rendering
3
Face integral projections

• Definition. Let i(x,y) be an image, and R(i) a
  region in it
• Vertical Integral Projection Horizontal
  Integral Projection
• PVR ymin, ..., ymax ? R PHR xmin, ...,
  xmax ? R
• PVR(y) i(x,y) ? (x,y) ? R(i) PHR(x)
  i(x,y) ? (x,y) ? R(i)

• When applied to human faces, typical patterns of
  projection appear.
• This is used to design a face detector and
  tracker.

Pv(y) Vertical I.P. of the face
Ph1(y) Horizon-tal I.P. of eyes region
Ph2(y) Horizon-tal I.P. of mouths region
4
Face tracking with I.P.
Face tracking is a 3-step process. 1.
Vertical alignment compute move-ment and scale
in vertical direction 2. Horizontal
alignment compute move-ment and scale in
horizontal direct. 3. Orientation alignment
compute orienta-tion of the face
PVFACE(y)
Align PVFACE
FACE
STEP 1
y
y
Align PHEYES
PHEYES(x)
EYES
STEP 2
x
x
Align PVEYEi
PVEYE1,PVEYE2
EYE1, EYE2
STEP 3
y
y

• Green line Projection model Red line
  Projection instance

5
Pose estimation (i)

• Depth estimation inversely proportional to the
  size of the head in the image. Approximated with
  the eye-to-eye distance.

Relative distances
0.674 0.947 1.184 1.636

• Roll estimation approximated with the perceived
  angle of the eyes.

Roll angles
-33.4º -13.0º 13.5º 20.2º
6
Pose estimation (ii)

• Pitch estimation a heuristic measure is defined,
  using the vertical I.P. of the face

Estimated pitch
19 -2 -9

• Yaw estimation another heuristic, using the
  horizontal I.P. of the eyes.

Estimated yaw
-7 0 11
7
Virtual environment

• The perceptual interface is used to control the
  movement in a virtual 3D world.
• We have used DirectX 9 and OpenCV 3.

Virtual 3D world
8
Results and conclusion

• Sample videos available athttp//dis.um.es/gine
  sgm/fip/percint.html

• Conclusions
• Depth and roll estimation is very reliable.
• Pitch and yaw are less stable.

????? ?.?. ?????,10/2004