Ongoing Challenges in Face Recognition

1
Ongoing Challenges in Face Recognition

• Peter Belhumeur
• Columbia University
• New York City

2
How are people identified?

• People are identified by three basic means
• Something they have (identity document or token)
• Something they know (password, PIN)
• Something they are (human body)

3
Iris
4
Retina
Every eye has its own totally unique pattern of
blood vessels.
5
Hand
6
Fingerprint
7
Ear
8
Face
9
Who are these people?
Sinha and Poggio 1996
10
Who are these people?
Sinha and Poggio 2002
11
Images as Points in Euclidean Space

• Let an n-pixel image to be a point in an n-D
  space, x ? Rn.
• Each pixel value is a coordinate of x.

12
Face Recognition Euclidean Distances
D1 gt 0
D2 gt 0

D3 0
13
Face Recognition Euclidean Distances
D1 gt 0
D2 gt 0
D3 gt D1 or 2
Hallinan 1994 Adini, Moses, and Ullman 1994
14

• Same Person
• or
• Different People

15

16

• Same Person
• or
• Different People

17
(No Transcript)
18
Why is Face Recognition Hard?
19
Challenges Image Variability
Short Term
Expression


Illumination


Pose
20
Illumination Invariants?
Does there exist a function f s.t.
f ( ) f ( ) f ( ) a
and
f ( ) f ( ) f ( ) b
?
21
Can Any Two Images Arise from a Single Surface?
n
s
a
I(x,y) a(x,y) n(x,y) s
I(x,y)
n
Same Albedo and Surface
Different Lighting
l
a
J(x,y) a(x,y) n(x,y) l
J(x,y)
22
The Surface PDE
I(x,y) a(x,y) n(x,y) s
( I l J s ) n 0
J(x,y) a(x,y) n(x,y) l
Linear PDE
Nonlinear PDE
23
Non-Existence Theorem for Illumination Invariants
Illumination invariants for 3-D objects do not
exist. This result does not ignore attached and
cast shadows, as well as surface interreflection.
Chen, Belhumeur, and Jacobs 2000
24
Geometric Invariants?
Does there exist a function f s.t.
f ( ) f ( ) f ( ) a
and
f ( ) f ( ) f ( ) b
?
25
Non-Existence Theorem for Geometric Invariants
Geometric invariants for rigid transformations of
3-D objects viewed under perspective projective
projection do not exist.
Burns, Weiss, and Riseman 1992
26
Image Variability Appearance Manifolds
x2
xn
x1
Lighting x Pose
Murase and Nayar 1993
27
Modeling Image Variability

• Can we model illumination and pose variability in
  images of a face?
• Yes, if we can determine the shape and texture of
  the face. But how?

28
Modeling Image Variability 3-D Faces

• Laser Range Scanners
• Stereo Cameras
• Structured Light
• Photometric Stereo

Atick, Griffin, Redlich 1996 Georghiades,
Belhumeur, Kriegman 1996 Blanz and Vetter 1999
Zhao and Chellepa 1999 Kimmel and Sapiro 2003
Geometrix 2001 MERL 2005
29
Illumination Variation Reveals Object Shape
n
s2
s1
a
s3
I2
I1
I3
Woodham 1984
30
Illumination Movie
Illumination Movie
31
Shape Movie
Shape Movie
32
Image Variability From Few to Many
Lighting x Pose
x2
xn
x1
Real
Synthetic
Georghiades, Belhumeur, and Kriegman 1999
33
Illumination Dome
34
Real vs. Synthetic
Synthetic
Real
35
Real vs. Synthetic
Real
Synthetic
36
A Step Back in Time
37
Albrecht Dürer, Four Books on Human Proportion
(1528)
38
Darcy Thompson, On Growth and Form (1917)
39
Darcy Thompson, On Growth and Form (1917)
40
Darcy Thompson, On Growth and Form (1917)
41
But what if we could .?
Blanz and Vetter 1999, 2003
42
Building a Morphable Face Model
Blanz and Vetter 1999, 2003
43
3-D Morphaple Models Semi-Automatic
Blanz and Vetter 1999, 2003
44
Building Morphable Face Models
Blanz and Vetter 1999, 2003
45
Fitting Morphable Face Models
Blanz and Vetter 1999, 2003