CS 395/495-26: Spring 2004

1
CS 395/495-26 Spring 2004

• IBMR Singular Value Decomposition(SVD Review)
• Jack Tumblin
• jet_at_cs.northwestern.edu

2
Matrix Multiply A Change-of-Axes

• Matrix Multiply Ax b
• x and b are column vectors
• A has m rows, n columns

x1 xn
a11 a1n am1 amn
b1 bm

b2
x2
x
b
Axb
b1
b3
x1
Input (N dim) (2D)
Output (M-dim) (3D)
3
Matrix Multiply A Change-of-Axes

• Matrix Multiply Ax b
• x and b are column vectors
• A has m rows, n columns
• Matrix multiply is just a set of dot-productsit
  changes coordinates for vector x, makes b.
• Rows of A A1,A2,A3,... new coordinate axes
• Ax a dot product for each new axis

x1 xn
a11 a1n am1 amn
b1 bm

A2
b2
x2
x
b
Axb
b1
A1
A3
b3
x1
Input (N dim) (2D)
Output (M-dim) (3D)
4
How Does Matrix stretch space?

• Sphere of all unit-length x ? ellipsoid of b
• Output ellipsoids axes are always perpendicular
  (true for any ellipsoid)
• THUS we can make -, unit-length axes

ellipsoid (disk)
b2
x2
b
x
Axb
b1
1
1
x1
b3
u2
u1
Input (N dim.)
Output (M dim.)
5
SVD finds Output Vectors U, and

• Sphere of all unit-length x ? ellipsoid of b
• Output ellipsoids axes form orthonormal
  basis vectors Ui
• Basis vectors Ui are columns of U matrix
• SVD(A) USVT columns of U OUTPUT basis
  vectors

ellipsoid (disk)
b2
x2
b
x
Axb
b1
1
1
x1
b3
u2
u1
Input (N dim.)
Output (M dim.)
6
SVD finds Input Vectors V, and

• For each Ui , make a matching Vi that
• Transforms to Ui (with scaling si) si (A Vi)
  Ui
• Forms an orthonormal basis of input space
• SVD(A) USVT columns of V INPUT basis
  vectors

b2
x2
x
b
v1
Axb
b1
v2
x1
b3
u2
u1
Input (N dim.)
Output (M dim.)
7
SVD finds Scale factors S.

• We have Unit-length Output Ui, vectors,
• Each from a Unit-length Input Vi vector, so
• So we need scale factor (singular values) si
  to link input to output si (A Vi) Ui
• SVD(A) USVT

s1
b2
x2
x
b
v1
Axb
b1
v2
x1
b3
s2
u2
u1
Input (N dim.)
Output (M dim.)
8
SVD Review What is it?

• Finish SVD(A) USVT
• add missing Ui or Vi, define these si0.
• Singular matrix S diagonals are si v-to-u
  scale factors
• Matrix U, Matrix V have columns Ui and Vi

b2
x2
x
b
u3
v1
Axb
b1
v2
x1
b3
u2
u1
Input (N dim.)
Output (M dim.)
A USVT Find Input Output Axes, Linked
by Scale
9
Let SVDs Explain it All for You

• cool! U and V are Orthonormal! U-1 UT, V-1
  VT
• Rank(A)? of non-zero singular values si
• ill conditioned? Some si are nearly zero
• Invert a non-square matrix A ? Amazing!
  pseudo-inverse AUSVT A VS-1UT I
  so A-1 V S-1 UT

b2
x2
x
u3
v1
Axb
b1
v2
x1
b3
u2
u1
Input (N dim.)
Output (M dim.)
A USVT Find Input Output Axes, Linked
by Scale
10
Solve for the Null Space Ax0

• Easy! x is the Vi axis that doesnt affect the
  output (si 0)
• are all si nonzero? then the only answer is x0.
• Null space because Vi is a DIMENSION--- x Vi
  a
• More than 1 zero-valued si? null space gt1
  dimension... x aVi1 bVi2

b2
x2
x
u3
v1
Axb
b1
v2
x1
b3
u2
u1
Input (N dim.)
Output (M dim.)
A USVT Find Input Output Axes, Linked
by Scale
11
More on SVDs Handout

• Zisserman Appendix 3, pg 556 (cryptic)
• Linear Algebra books
• Good online introduction (your handout) Leach,
  S. Singular Value Decomposition A primer,
  Unpublished Manuscript, Department of Computer
  Science, Brown University. http//www.cs.brown.edu
  /research/ai/dynamics/tutorial/Postscript/
• Google on Singular Value Decomposition...

12
END