Positive definite matrices

1
Positive definite matrices Quadratic form of a
positive definite matrix. Given the following
function f, defined with a symmetric matrix M and
inputs x
If f is always positive irrespective of x then M
is positive definite. If x is a two valued vector
x1 x2 , then f looks like a bowl resting on
the origin as seen in the figure.
T
2
Two other shapes can result from the quadratic
form. If f is always negative then M is known as
negative definite. If f is positive in some
regions and negative on others then it describes
a saddle. In all cases f is zero when x0
3
Theorem If a matrix
then it is positive definite Proof. The
quadratic form is If we can show that f is
always positive then M must be positive definite.
We can write this as Provided that Ux always
gives a non zero vector for all values of x
except when x0 we can write b U x, i.e. so
f must always be positive
4
Least squares linear regression Courtesy of
Johann Fredrich Carl Gauss Given a model and some
data, we wish to calculate the best
coefficients. One way is to minimise the errors.
5
Consider a model of the form (first model)
We can write this in matrix form by putting all
the data into a model output vector , the
parameters into a vector
6
Consider a model of the form (second model)
The parameters vector The model output vector
7
The actual output vector
A model can be estimated by using as a
target of
y --- actual output (measurements) --- the
output that is expected by the model
The model parameter ? is derived so that the
distance between these two vectors is the
smallest possible.
8
In vector form
9
Rearrange this as follows
1. Only the last term (which is in quadratic
form) contains ?, 2.
is positive definite.
10
SSE has the minimum (best fit)
at
(the hat means estimate)
is the solution of least square parameter
estimate .
11
An alternative proof