Singular Value Decomposition and Numerical Rank

1
Singular Value Decomposition and Numerical Rank
2

• The SVD was established
• for real square matrices in the
  1870s by Beltrami Jordan
• for complex square matrices by
  Autonne
• for general rectangular matrices by
  Eckart Young
• (autonne-Eckart-Young theorem)
• Theorem
• Let . Then there exist
  orthogonal unitary matrices
• and
  such that
• where
• and
  with

3

• Since , we have
  . Denoting by
• we can arrange
  that
  . Let be a
  corresponding set of orthonormal eigenvectors and
  let
• Then if we
  have
• where
  
• Also so that
• and thus
• Let . Then from we
  have
• Choose any such that
  is orthogonal.
• Then
• and so as desired.

4

• The numbers together with
  are called the
  singular values of and they are positive
  square roots of the eigenvalues (which are non
  negative) of .
• The columns of are called the left singular
  vector of (the orthonormal eigenvectors of
  ) while the columns of
  are called the right singular vector of
  (the orthonormal eigenvectors of
  ).
• The matrix has singular
  values, the positive square roots of the
  eigenvalues of . The
• nonzero singular
  values of and are the same.

5

• It is not generally a good idea to compute the
  singular values of by the first finding the
  eigenvalues of , tempting as that is.
• Ex Let be a real number with
  (so that )
• Let
• Then
• so we compute
  leading to the (erroneous) conclusion that the
  rank of is 1.
• If we could compute in infinite precision, we
  would have
• with and thus
  .
• The point is that by working with we
  have unnecessarily introduced into the
  computation.

6

• It is clear from the definition that the number
  of nonzero singular values of determines
  its rank while the question is not nearly
  clear-cut in the context of computation on a
  digital computer, it is now generally
  acknowledged that the singular value
  decomposition is the only generally reliable
  method of determining rank numerically. ---------
  look at the smallest non-zero singular value of
  a matrix .
• Since that computed value is exact for a matrix
  near , it makes sense to consider the rank of
  all matrices in some -ball (w,r,t. the
  spectral norm say) around .
• The choice of may also be based on
  measurement errors incurred in estimating the
  coefficients of or the coefficients may be
  uncertain because of round off errors incurred in
  a previous computation to get them.
• The key quantity in rank determination is .

7

• The smallest nonzero singular value gives a
  dependable measure of how far (in the
  sense) a matrix is from matrices of lesser rank.
• But alone is clearly sensitive to scale so
  that a better measure
• is . But so the
  important quantity is which
• turns out to be the reciprocal of the number
  , the so-called
  conditional number of A w.r.t. pseudo inversion.
• In the case when A is invertible,
  is usual spectral condition number
  w.r.t. inversion.
• Ref Stewart On the pertubation of
  pseudo-inverses, projections, and linear least
  squares problems, SIAM Review, vol.19,
  pp.634-662, 1977.
• In solving the linear system , the
  condition number
• gives a measure of
  how much errors in A and/or may be
  magnified in the computed solution. Moreover, if
  , gives a measure of the
  nearness of A to singularity.

8

• In fact measures the nearness of A
  to singularity in any matrix norm and
  for certain norms it is easy to construct
  explicitly a matrix E with
• and AE
  singular.
• Ex Consider the matrix
• is, in fact, very near singular and gets
  more nearly so as n increases.
• Adding to every element in the
  first column of A gives an exactly singular
  matrix.
• Rank determination, in the presence of round off
  error, is a highly nontrivial problem.