Conjugate Gradient

1
Conjugate Gradient
2
0. History

• Why iterate?
• Direct algorithms require O(n³) work.
• 1950 n20
• 1965 n200
• 1980 n2000
• 1995 n20000

dimensional increase 103 computer hardware 109
3
0. History

• If matrix problems could be solved in O(n²) time,
  matrices could be 30 bigger.
• There are direct algorithms that run in about
  O(n2.4) time, but their constant factors are to
  big for practicle use.
• For certain matrices, iterative methods have the
  potential to reduce computation time to O(m²).

4
1. Introduction

• CG is the most popular method for solving large
  systems of linear equations Ax b.
• CG is an iterative method, suited for use with
  sparse matrices with certain properties.
• In practise, we generally dont find dense
  matrices of a huge dimension, since the huge
  matrices often arise from discretisation of
  differential of integral equations.

5
2. Notation

• Matrix A, with components Aij
• Vector, n x 1 matrix x, with components xi
• Linear equation Axbwith components S Aij xj
  bi

6
2. Notation

• Transponation of a matirx (AT)ij Aji
• Inner product of two vectors xTy S xiyi
• If xTy 0, then x and y are orthogonal

7
3. Properties of A

• A has to be an n x n matrix.
• A has to be positive definite, xTAx gt 0
• A has to be symmetric, AT A

8
4. Quadratic Forms

• A QF is a scalar quadratic function of a vector
• Example

9
4. Quadratic Forms

• Gradient Points to the greatest increase of f(x)

10
4. Quadratic Forms
positive definite xT A x gt 0
negative definite xT A x lt 0
positive indefinite xT A x 0
indefinite
11
5. Steepest Descent

• Start at an arbitrary point and slide down to the
  bottom of the paraboloid.
• Steps x(1), x(2), in the direction f(xi)
• Error e(i) x(i) x
• Residual r(i) b Ax(i)

r(i) - Ae(i) r(i) - f(x(i))
12
5. Steepest Descent

• x(i1) x(i) a r(i) , but how big is a?
• f(x(i1)) orthogonal to r(i)

search line a r(i)
13
5. Steepest Descent
The algorithm above requires two matrix
multiplications per iteration. One can be
eliminated by multiplying the last equation by
A.
14
5. Steepest Descent
This sequence is generated without any feedback
of x(i). Therefore, floatingpoint roundoff errors
may accumulate and the sequence could converge at
some point near x. This effect can be avoided by
periodically recomputing the correct residual
using x(i).
15
6. Eigenvectors

• v is an eigenvector of A, if a scalar ? so that
  A v ? v
• ? is then called an eigenvalue.
• A symmetric n x n matrix always has n independent
  eigenvectors which are orthogonal.
• A positive definite matrix has positive
  eigenvalues.

16
7. Convergence of SD

• Convergence of SD requires the error e(i) to
  vanish. To measure e(i), we use the A- norm
• Some math now yields

17
7. Convergence of SD
Spectral condition number
.
An upper bound for ? is found by setting
.
We therefore have instant convergence if all the
eigenvalues of A are the same.
18
7. Convergence of SD
large ? small µ
large ? large µ
small ? large µ
small ? small µ
19
8. Conjugate Directions

• Steepest Descent often takes steps in the same
  direction as earlier steps.
• The solution is to take a set of A-orthogonal
  search directions d(0), d(1), , d(n-1) and take
  exactly one step of the right length in each
  direction.

20
8. Conjugate Directions
A-orthogonal
orthogonal
21
8. Conjugate Directions

• Demanding dT(i) to be A-orthogonal on the next
  error e(i1), we get
  .
• Generating search directions by Gram-Schmidt
  Conjugation. Problem O (n³)

22
8. Conjugate Directions

• CD chooses , so thatis
  minimized.
• The error term is therefore A-orthogonal to all
  the old search directions.

23
9. Conjugate Gradient

• The residual is orthogonal to the previous search
  directions.
• Krylov subspace

24
9. Conjugate Gradient
r(i1) is A-orthogonal to Di

• Gram-Schmidt conjugation becomes easy, because
  r(i1) is already A-orthogonal to all the
  previous search directions except d(i).

25
9. Conjugate Gradient
26
11. Preconditioning

• Improving the condition number of the matrix
  before the calculation. Example
• Attempt to strech the quadratic form to make it
  more spherical.
• Many more sophisticated preconditioners have been
  developed and are nearly always used.

27
12. Outlook

• CG can also be used to solve
• To solve non-linear Problems with CG, one has to
  make changes in the algorithm. There are several
  possibilities, and the best choice is still under
  research.

.
28
12. Outlook
In non-linear problems, there may be several
local minima to which CG might converge. It is
therefore hard to determine the right step size.
29
12. Outlook

• There are other algorithms in numerical linear
  algebra closely related to CG.
• They all use Krylov subspaces.