A Primer to Singular Value Decomposition SVD

1
A Primer to Singular Value Decomposition (SVD)

• Chih-Chieh Hung
• 2005.4.26

2
Outline

• Linear Algebra
• SVD
• Applications of SVD

3
Linear Algebra

• Relative Conception
• Matrix Actions

4
Relative Conceptions

• Orthogonal Matrix
• Orthonormal
• Eigenvector Eigenvalue

5
Matrix Actions

• Matrix Hits
• When you hit a point with a matrix, you got
  another point.
• Example

6
Some Examples for Matrix Action

• when the matrix  hits the points on the
  unit circle

7
Perpframe

• A perpframe of n dimensions consists of n
  mutually perpendicular unit vectors.
• Ex (2D perpframe)

8
Hanger

• The hanger matrix hangs the xy-axis onto the
  perpframe
• Ex
• perpframe consists of
• hanger

9
Aligner

• The aligner matrix aligns the perpframe to the
  xy-axis.
• Ex
• perpframe consists of
• Aligner

10
Stretcher

• The diagonal matrix  stretches in the x
  direction by a factor of "a" and in the y
  direction by a factor of "b". 
• Ex

11
Outline

• Linear Algebra
• SVD
• Applications of SVD

12
Singular Value Decomposition

• The singular value decomposition for a matrix A
  writes A as a product (hanger)(stretcher)(aligner)
  .
• Two-Third Theorem
• For an  mn matrix A Rn?Rm and any
  orthonormal basis of Rn,
  define and
• Then

13
Intuition of SVD

• Aligner
• Stretcher
• Hanger

14
Formula For SVD
15
Outline

• Linear Algebra
• SVD
• Applications of SVD

16
Applications of SVD

• Linear System Problem
• Data Compression
• Latent Semantic Indexing (LSI)

17
Linear System Problem

• Suppose the SVD for a matrix is
• How can you use the decomposition to solve the
  matrix equation Ax b ?

18
Pseudo Inverse

• The following matrix is called pseudo inverse
• If Ax b has solution, then A A-1 (i.e. x
  Ab)
• If Ax b has no solution, then x Ab is the
  least square approximation.

Vk Sk UTk
19
Data Compression

• Suppose that the image is stored as a 256 x 264
  matrix M with entries between 0 and 1.
• We use SVD to approximate the original image 

20
Reduced SVD
21
Evaluation of Data Compress by SVD

• To send the matrix M you need to send 256 x 264
  67584 numbers. 
• To send the rank 36 approximation to M you need
  only send
• the first 36 singular values, 
• the first 36 hanger vectors, each of which has
  256 entries, 
• the first 36 aligner vectors, each of which has
  264 entries. 
• So in total you need to send only
  36(1256264)18756 numbers.

264
M
256
22
Latent Semantic Indexing(LSI)

• Uses linear algebra technique called singular
  value decomposition (SVD)
• attempts to estimate the hidden structure
• discovers the most important associative patterns
  between words and concepts

23
An Example (1/7)
24
An Example (2/7)
25
An Example (3/7)
(U1s1, U2s2)
(V1s1, V2s2)
26
An Example (4/7)

• Query

application
theory
27
An Example (6/7)
28
An Example (7/7)

• Lexical Matching B3, B11, B12, B17
• LSI approach B3, B11, B12, B17, B5, B6, B7, B16