Template-Based Face Detection

1

• Lecture 7
• Template-Based Face Detection
• Principal Component Analysis (part 1)

CSE 6367 Computer Vision Spring 2010 Vassilis
Athitsos University of Texas at Arlington
2
Face Detection

• We will make a few assumptions, to simplify the
  problem
• The face is fully visible.

Example cases where the face is not fully visible
3
Face Detection

• We will make a few assumptions, to simplify the
  problem
• The face is fully visible.
• We see a frontal view of the face, i.e., the face
  is facing towards the camera.

Example cases where the view of the face is not
frontal
4
Face Detection

• We will make a few assumptions, to simplify the
  problem
• The face is fully visible.
• We see a frontal view of the face, i.e., the face
  is facing towards the camera.
• The face is more or less in an upright
  orientation.

Example of a face not in an upright orientation
5
What is a Template?

• A template is an example of how an object looks.
• What example would be appropriate if we are
  looking for a face?

6
What is a Template?

• A template is an example of how an object looks.
• What example would be appropriate if we are
  looking for a face?
• An average face.

7
Computing an Average Face

• We need aligned face images
• In this case, aligned means
• same size
• significant features (eyes, nose, mouth), to the
  degree possible, are in similar pixel locations.

an example set of aligned face images
8
Computing an Average Face
Note that filenames is a cell variable (look it
up!). Using cells is a good way to define a set
of strings. filenames '4846d101.bmp' '4848d101
.bmp' '4851d101.bmp' '4853d101.bmp' '4854d101.bmp'
number prod(size(filenames)) image_vertica
l 100 image_horizontal 100 total
zeros(image_vertical, image_horizontal) for
index 1 number image read_gray(filenames
index) total total image
disp(index) end average_face total / number
9
Result Average Face
10
Defining a Face Template

• Keep the parts of the average face that are most
  likely to be present in all faces
• Exclude background.
• Exclude forehead (highly variable appearance, due
  to hair).
• Exclude lower chin.

face template
average face
11
Using a Template to Find Faces

• How can we use a face template to perform face
  detection in an image?

12
Finding Matches for the Template

• How can we find good matches for a given template?

13
Using Normalized Correlation
function result vector_normalized_correlation(v1
, v2) function result vector_normalized_corr
elation(v1, v2) centered_v1 v1 -
mean(v1) centered_v2 v2 - mean(v2) norm1
norm(centered_v1) norm2 norm(centered_v2)
result centered_v1 . centered_v2 / (norm1
norm2)

• Function normxcorr2(template, image) returns a
  matrix of normalized correlation scores between
  the template and each template-sized subwindow of
  the image.

14
Invoking normxcorr2
photo read_gray('vassilis1g.bmp') result
normxcorr2(face_filter, photo)
photo
result
result gt 0.6

• It found the face!
• The result of normxcorr2 has larger size than the
  input.
• The face in photo matched the scale of the
  template.

15
normalized_correlation

• A wrapper around normxcorr2.
• The result has the same size as the image.
• Border values are zero.

16
function result normalized_correlation(image,
template) function result
normalized_correlation(image, template)
Returns a matrix containing normalized
correlation results between template and all
template-sized subwindows of image.
image_rows, image_columns size(
image) template_rows, template_columns
size(template) row_start floor(template_rows /
2) 1 row_end row_start image_rows -
1 col_start floor(template_columns / 2)
1 col_end col_start image_columns - 1
result normxcorr2(template, image) result_row
s, result_columns size(result) result(1templa
te_rows, ) 0 result((result_rows-template_rows
1)result_rows, ) 0 result(,
1template_columns) 0 result(,
(result_columns-template_columns1)result_rows,
) 0 result result(row_startrow_end,
col_startcol_end)
17
Invoking normalized_correlation
photo read_gray('vassilis1g.bmp') result
normalized_correlation(photo, face_filter)
18
A Trick for Visualizing Results
photo read_gray('vassilis1g.bmp') result
normalized_correlation(photo, face_filter) visual
ization max((result gt 0.6)255, photo 0.7)
result
visualization
photo
19
Problem Different Scales
photo read_gray('vassilis1e.bmp') result
normalized_correlation(photo, face_filter) visual
ization max((result gt 0.35)255, photo 0.7)
visualization
photo
result
20
Solution Multiscale Search
function ... result, max_scales
multiscale_correlation(image, template, scales)
function result multiscale_correlation(image,
template, scales) for each pixel, search
over the specified scales, and record - in
result, the max normalized correlation score for
that pixel over all scales - in max_scales,
the scale that gave the max score
21
Solution Multiscale Search
function ... result, max_scales
multiscale_correlation(image, template, scales)
result ones(size(image)) -10 max_scales
ones(size(image)) -10 for scale scales
for efficiency, we either downsize the image,
or the template, depending on the current
scale if scale gt 1 scaled_image
imresize(image, 1/scale, 'bilinear')
temp_result normalized_correlation(scaled_image,
template) temp_result
imresize(temp_result, size(image), 'bilinear')
else scaled_image image
scaled_template imresize(template, scale,
'bilinear') temp_result
normalized_correlation(image, scaled_template)
end higher_maxes (temp_result gt
result) max_scales(higher_maxes) scale
result(higher_maxes) temp_result(higher_maxes)
end
22
Results of multiscale_correlation
photo read_gray('vassilis1e.bmp') scales
0.50.13 result2, max_scales ...
multiscale_correlation(photo, face_filter,
scales) visualization2 max((result2 gt
0.6)255, photo 0.7)
visualization2
photo
result2
23
Handling Rotations
load face_filter photo read_gray('vassilis2b.bm
p') result, boxes ...
template_detector_demo(photo, face_filter,
0.50.13., 0, 1)
photo
result
24
Handling Rotations
load face_filter photo read_gray('vassilis2b.bm
p') resultb, boxes ...
template_detector_demo(photo, face_filter,
0.50.13., 0, 2)
photo
resultb
25
Code for Template Search

• Useful functions
• template_search
• find_template
• template_detector_demo

26
function max_responses, max_scales,
max_rotations ... template_search(image,
template, scales, rotations, result_number)
function result, max_scales, max_rotations
... template_search(image, template, scales,
rotations, result_number) for each pixel,
search over the specified scales and rotations,
and record - in result, the max normalized
correlation score for that pixel over all
scales - in max_scales, the scale that gave the
max score - in max_rotations, the rotation
that gave the max score clockwise rotations
are positive, counterclockwise rotations are
negative. rotations are specified in degrees
27
function max_responses, max_scales,
max_rotations ... template_search(image,
template, scales, rotations, result_number)
max_responses ones(size(image))
-10 max_scales zeros(size(image)) max_rotation
s zeros(size(image)) for rotation
rotations rotated imrotate(image,
-rotation, 'bilinear', 'crop') responses,
temp_max_scales ...
multiscale_correlation(rotated, template,
scales) responses imrotate(responses,
rotation, 'nearest', 'crop') temp_max_scales
imrotate(temp_max_scales, rotation, ...
'nearest', 'crop')
higher_maxes (responses gt max_responses)
max_responses(higher_maxes) responses(higher_max
es) max_scales(higher_maxes)
temp_max_scales(higher_maxes)
max_rotations(higher_maxes) rotation end
28
function result, boxes ...
template_detector_demo(image, template, ...
scales, rotations,
result_number) function result, boxes
template_detector_demo(image, template,
... scales,
rotations, result_number) returns an image
that is a copy of the input image, with the
bounding boxes drawn for each of the best matches
for the template in the image, after searching
all specified scales and rotations. boxes
find_template(image, template, scales, ...
rotations, result_number) result
image for number 1result_number
result draw_rectangle1(result, boxes(number,
1), ...
boxes(number, 2), ...
boxes(number, 3), boxes(number, 4)) end
29
Handling Rotations
load face_filter photo read_gray('vassilis2b.bm
p') resultb, boxes ...
template_detector_demo(photo, face_filter,
0.50.13., 0, 2)
photo
resultb
30
Handling Rotations
load face_filter photo read_gray('vassilis2b.bm
p') scales 0.50.13 rotations
-10510 result2, boxes template_detector_dem
o(photo, face_filter, ...,
scales, rotations, 1)
photo
result2
31
Principal Component Analysis
32
Vector Spaces

• For our purposes, a vector is a tuple of d
  numbers X (x1, x2, ..., xd).
• An example vector space the 2D plane.
• Every point in the plot is a vector.
• Specified by two numbers.

33
Images are Vectors

• An M-row x N-column image is a vector of
  dimensionality ...

34
Images are Vectors

• An M-row x N-column image is a vector of
  dimensionality.
• MN if the image is grayscale.
• MN3 if the image is in color.

35
Images are Vectors

• Consider a 4x3 grayscale image
• A A11 A12 A13 A21 A22 A23 A31 A32
  A33 A41 A42 A43
• The (Matlab) vector representation Av of A isAv
  A11 A21 A31 A41 A12 A22 A32 A42 A13 A23 A33
  A43
• Mathematically, the choice of how to order does
  not matter, AS LONG AS IT IS CONSISTENT FOR ALL
  IMAGES.
• In Matlab, to vectorize an image
• Av A()
• NOTE The above returns a COLUMN vector.

36
Vector Operations Addition

• (x1, ..., xd) (y1, ..., yd) (x1y1, ...,
  xdyd)
• Example in 2D
• A (-3, 2)
• B (1, 4)
• AB (-1, 6).

AB
B
A
37
Vector Operations Addition

• (x1, ..., xd) (y1, ..., yd) (x1y1, ...,
  xdyd)
• Example in 2D
• A (-3, 2)
• B (1, 4)
• AB (-1, 6).

AB
B
A
38
Addition in Image Space


MxN imageall pixels (0, 255, 0)
MxN imageall pixels (255, 0, 0)


39
Addition in Image Space


MxN imageall pixels (0, 255, 0)
MxN imageall pixels (255, 255, 0)
MxN imageall pixels (255, 0, 0)


What is the rangeof pixel values here?
40
Addition in Image Space


MxN imageall pixels (0, 255, 0)
MxN imageall pixels (255, 255, 0)
MxN imageall pixels (255, 0, 0)


Range of pixel valuesfrom 0 to 2255
41
Vector Operations Scalar Multiplication

• c (x1, ..., xd) (c x1, ..., c xd)
• (x1, ..., xd) / c 1/c (x1, ..., xd)
  (x1/c, ..., xd/c)
• Example in 2D
• A (-3, 2)
• 3 A (-9, 6)
• A / 3 (-1, 0.67).

3A
A
A / 3
42
Multiplication in Image Space
image
image 0.7
image 0.5
43
Operations in Image Space

• Note with addition and multiplication we can
  easily generate images with values outside the
  0, 255 range.
• These operations are perfectly legal.
• However, we cannot visualize the results
  directly.
• Must convert back to 0 255 range to visualize.

44
Linear Combinations

• Example c1 v1 c2 v2 c3 v3
• c1, c2, c3 real numbers.
• v1, v2, v3 vectors of d dimensions.
• What is the type of the result?

45
Linear Combinations

• Example c1 v1 c2 v2 c3 v3
• c1, c2, c3 real numbers.
• v1, v2, v3 vectors of d dimensions.
• result vector of d dimensions.

46
Vectors Must Be of Same Size

• We cannot add vectors that are not of the same
  size.
• (1,2) (0,1,0) is NOT DEFINED.
• To add images A and B
• IMPORTANT Most of the time, it only makes sense
  to add A and B ONLY if they have the same number
  of rows and the same number of columns.
• WARNING Matlab will happily do the following

a rand(4,3) b rand(6,2) c a() b()
47
Example Linear Combination
a
b
c
d
e
a read_gray('4309d111.bmp') b
read_gray('4286d201.bmp') c read_gray('4846d101
.bmp') d read_gray('4848d101.bmp') e
read_gray('4853d101.bmp') avg 0.2a 0.2b
0.2c 0.2d 0.2e or, equivalently avg
(abcde) / 5
avg
48
Dimensionality Reduction

• Consider a set of vectors in a d-dimensional
  space.
• How many numbers do we need to represent each
  vector?

49
Dimensionality Reduction

• Consider a set of vectors in a d-dimensional
  space.
• How many numbers do we need to represent each
  vector?
• At most d the same as the number of dimensions.
• Can we use fewer?

50
Dimensionality Reduction

• Consider a set of vectors in a d-dimensional
  space.
• How many numbers do we need to represent each
  vector?
• At most d the same as the number of dimensions.
• Can we use fewer?

51
Dimensionality Reduction

• In this example, every point (x, y) is on a line
• y ax b
• If we have 100 points on this plot, how many
  numbers do we need to specify them?

52
Dimensionality Reduction

• In this example, every point (x, y) is on a line
• y ax b
• If we have 100 points on this plot, how many
  numbers do we need to specify them?
• 102 a, b, and the x coordinate of each point.
• Asymptotically one number per point.

53
Lossy Dimensionality Reduction

• Suppose we want to project all points to a single
  line.
• This will be lossy.
• What would be the best line?

54
Lossy Dimensionality Reduction

• Suppose we want to project all points to a single
  line.
• This will be lossy.
• What would be the best line?
• Optimization problem.
• Infinite answers.
• We must define how to evaluate each answer.

55
Optimization Criterion

• We want to measure how good a projection P is,
  GIVEN A SET OF POINTS.
• If we dont have a specific set of points in
  mind, what would be the best projection?

56
Optimization Criterion

• We want to measure how good a projection P is,
  GIVEN A SET OF POINTS.
• If we dont have a specific set of points in
  mind, what would be the best projection?
• NONE all are equally good/bad.

57
Optimization Criterion

• Consider a pair of points X1, X2.
• D1 squared distance from X1 to X2.
• sum((X1X2) . (X1X2))
• D2 squared distance from P(X1) to P(X2).
• Error(X1, X2) D1 D2.
• Will it ever be negative?

X2
X1
P(X1)
P(X2)
58
Optimization Criterion

• Consider a pair of points X1, X2.
• D1 squared distance from X1 to X2.
• sum((X1X2) . (X1X2))
• D2 squared distance from P(X1) to P(X2).
• Error(X1, X2) D1 D2.
• Will it ever be negative?
• NO D1 gt D2 always.

X2
X1
P(X1)
P(X2)
59
Optimization Criterion

• Now, consider the entire set of points
• X1, X2, ..., Xn.
• Error(P) sum(Error(Xi, Xj) i, j 1, ..., n,
  i ! j).

• Interpretation
• We measure how well P preserves distances.
• If P preserves distances, Error(P) 0.

60
Example Perfect Projection Exists

• In this case, projecting to a line oriented at 30
  degrees would give zero error.

61
Finding the Best Projection PCA

• First step center the data.

number size(points, 2) note that we are
transposing twice average mean(points')' cent
ered_points zeros(size(points)) for index
1number centered_points(, index)
points(, index) - average end
plot_points(centered_points, 2)
62
Finding the Best Projection PCA

• First step center the data.

points
centered_points
63
Finding the Best Projection PCA

• Second step compute the covariance matrix.
• In the above line we assume that each column is a
  vector.

covariance_matrix centered_points
centered_points'
64
Finding the Best Projection PCA

• Second step compute the covariance matrix.
• In the above line we assume that each column is a
  vector.
• Third step compute the eigenvectors and
  eigenvalues of the covariance matrix.

covariance_matrix centered_points
centered_points'
eigenvectors eigenvalues eig(covariance_matrix
)
65
Eigenvectors and Eigenvalues
eigenvectors 0.4837 -0.8753 -0.8753
-0.4837 eigenvalues 2.0217 0
0 77.2183

• Each eigenvector v is a column, that specifies a
  line going through the origin.
• The importance of the i-th eigenvector is
  reflected by the i-th eigenvalue.
• second eigenvalue 77, first eigenvalue 2, gt
  second eigenvector is far more important.

66
Visualizing the Eigenvectors
black v1 (eigenvalue 2.02) red v2 (eigenvalue
77.2)
plot_points(points, 1) p1 eigenvectors(,
1) p2 eigenvectors(, 2) plot(0 p1(1), 0,
p1(2), 'k-', 'linewidth', 3) plot(0 p2(1),
0, p2(2), 'r-', 'linewidth', 3)
67
PCA Code
function average, eigenvectors, eigenvalues
...
compute_pca(vectors) number size(vectors,
2) note that we are transposing twice average
mean(vectors')' centered_vectors
zeros(size(vectors)) for index 1number
centered_vectors(, index) vectors(, index) -
average end covariance_matrix
centered_vectors centered_vectors' eigenvector
s eigenvalues eig( covariance_matrix)
eigenvalues is a matrix, but only the diagonal
matters, so we throw away the rest eigenvalues
diag(eigenvalues) eigenvalues, indices
sort(eigenvalues, 'descend') eigenvectors
eigenvectors(, indices)