Stats 346'3

1
Stats 346.3

• Multivariate Data Analysis

2
Multivariate Data

• We have collected data for each case in the
  sample or population on not just one variable but
  on several variables X1, X2, Xp
• This is likely the situation very rarely do you
  collect data on a single variable.
• The variables maybe
• Discrete (Categorical)
• Continuous (Numerical)
• The variables may be
• Dependent (Response variables)
• Independent (Predictor variables)

3
Multivariate Techniques

• Multivariate Techniques can be classified as
  follows
• Techniques that are direct analogues of
  univariate procedures.
• There are univariate techniques that are then
  generalized to the multivariate situarion
• e. g. The two independent sample t test,
  generalized to Hotellings T2 test
• ANOVA (Analysis of Variance) generalized to
  MANOVA (Multivariate Analysis of Variance)

4

• Techniques that are purely multivariate
  procedures.
• Correlation, Partial correlation, Multiple
  correlation, Canonical Correlation
• Principle component Analysis, Factor Analysis
• These are techniques for studying complicated
  correlation structure amongst a collection of
  variables

5

• Techniques for which a univariate procedures
  could exist but these techniques become much more
  interesting in the multivariate setting.
• Cluster Analysis and Classification
• Here we try to identify subpopulations from the
  data
• Discriminant Analysis
• In Discriminant Analysis, we attempt to use a
  collection of variables to identify the unknown
  population for which a case is a member

6
Cluster Analysis of n 132 university students
using responses from Meaning of Life
questionnaire (40 questions)
7
Discriminant Analysis of n 132 university
students into the three identified populations
8
A Review of Linear Algebra

• With some Additions

9
Matrix Algebra
Definition An n m matrix, A, is a rectangular
array of elements
n of columns m of rows dimensions n m
10
Definition A vector, v, of dimension n is an n
1 matrix rectangular array of elements
vectors will be column vectors (they may also be
row vectors)
11
A vector, v, of dimension n
can be thought a point in n dimensional space
12
v3
v2
v1
13
Matrix Operations
Addition Let A (aij) and B (bij) denote two n
m matrices Then the sum, A B, is the matrix
The dimensions of A and B are required to be both
n m.
14
Scalar Multiplication Let A (aij) denote an n
m matrix and let c be any scalar. Then cA is the
matrix
15
Addition for vectors
v3
v2
v1
16
Scalar Multiplication for vectors
v3
v2
v1
17
Matrix multiplication Let A (aij) denote an n
m matrix and B (bjl) denote an m k matrix
Then the n k matrix C (cil) where
is called the product of A and B and is denoted
by AB
18
In the case that A (aij) is an n m matrix and
B v (vj) is an m 1 vector Then w Av
(wi) where
is an n 1 vector
w3
v3
w2
v2
w1
v1
19
Definition An n n identity matrix, I, is the
square matrix
Note

• AI A
• IA A.

20
Definition (The inverse of an n n matrix)
Let A denote the n n matrix
Let B denote an n n matrix such that
AB BA I, If the matrix B exists then A is
called invertible Also B is called the inverse of
A and is denoted by A-1
21
The Woodbury Theorem
where the inverses
22
Proof Let
Then all we need to show is that H(A BCD)
(A BCD) H I.
23
(No Transcript)
24
The Woodbury theorem can be used to find the
inverse of some pattern matrices Example Find
the inverse of the n n matrix
25
where
hence
and
26
Thus
Now using the Woodbury theorem
27
Thus
28
where
29
Note for n 2
30
Also
31
Now
32
and
This verifies that we have calculated the inverse
33
Block Matrices
Let the n m matrix
be partitioned into sub-matrices A11, A12, A21,
A22,
Similarly partition the m k matrix
34
Product of Blocked Matrices
Then
35
The Inverse of Blocked Matrices
Let the n n matrix
be partitioned into sub-matrices A11, A12, A21,
A22,
Similarly partition the n n matrix
Suppose that B A-1
36
Product of Blocked Matrices
Then
37
Hence
From (1)
From (3)
38
Hence
or
using the Woodbury Theorem
Similarly
39
From
and
similarly
40
Summarizing
Let
Suppose that A-1 B
then
41
Example
Let
Find A-1 B
42
(No Transcript)
43
The transpose of a matrix
Consider the n m matrix, A
then the m n matrix, (also denoted by AT)
is called the transpose of A
44
Symmetric Matrices

• An n n matrix, A, is said to be symmetric if
• Note

45
The trace and the determinant of a square matrix
Let A denote then n n matrix
Then
46
also
where
47
Some properties
48
Some additional Linear Algebra
49
Inner product of vectors
Let denote two p 1 vectors.
Then.
50
Note
Let denote two p 1 vectors.
Then.
51
Note
Let denote two p 1 vectors.
Then.
0
52
Special Types of Matrices

• Orthogonal matrices
• A matrix is orthogonal if PP PP I
• In this cases P-1P .
• Also the rows (columns) of P have length 1 and
  are orthogonal to each other

53
Suppose P is an orthogonal matrix
then
Let denote p 1 vectors.
Orthogonal transformation preserve length and
angles Rotations about the origin, Reflections
54
Example
The following matrix P is orthogonal
55
Special Types of Matrices(continued)

• Positive definite matrices
• A symmetric matrix, A, is called positive
  definite if
• A symmetric matrix, A, is called positive semi
  definite if

56

• If the matrix A is positive definite then

57
Theorem The matrix A is positive definite if
58
Special Types of Matrices(continued)

• Idempotent matrices
• A symmetric matrix, E, is called idempotent if
• Idempotent matrices project vectors onto a linear
  subspace

59
Definition

• Let A be an n n matrix
• Let

then l is called an eigenvalue of A and
and is called an eigenvector of A and
60
Note
61
polynomial of degree n in l.
Hence there are n possible eigenvalues l1, , ln
62
Thereom If the matrix A is symmetric then the
eigenvalues of A, l1, , ln,are real.
Thereom If the matrix A is positive definite
then the eigenvalues of A, l1, , ln, are
positive.
Proof A is positive definite if
be an eigenvalue and
Let
corresponding eigenvector of A.
63
Thereom If the matrix A is symmetric and the
eigenvalues of A are l1, , ln, with
corresponding eigenvectors
If li ? lj then
Proof Note
64
Thereom If the matrix A is symmetric with
distinct eigenvalues, l1, , ln, with
corresponding eigenvectors
Assume
65
proof
Note
and
P is called an orthogonal matrix
66
therefore
thus
67

• Comment
• The previous result is also true if the
  eigenvalues are not distinct.
• Namely if the matrix A is symmetric with
  eigenvalues, l1, , ln, with corresponding
  eigenvectors of unit length

68
An algorithm for computing eigenvectors,
eigenvalues of positive definite matrices

• Generally to compute eigenvalues of a matrix we
  need to first solve the equation for all values
  of l.
• A lI 0 (a polynomial of degree n in l)
• Then solve the equation for the eigenvector

69
Recall that if A is positive definite then
It can be shown that
and that
70
Thus for large values of m

• The algorithim
• Compute powers of A - A2 , A4 , A8 , A16 , ...
• Rescale (so that largest element is 1 (say))
• Continue until there is no change, The resulting
  matrix will be
• Find
• Find

71
To find

• Repeat steps 1 to 5 with the above matrix to find
• Continue to find

72
Example

• A

73
Differentiation with respect to a vector, matrix
74
Differentiation with respect to a vector
Let denote a p 1 vector. Let
denote a function of the components of .
75
Rules
1. Suppose
76
2. Suppose
77
Example
1. Determine when
is a maximum or minimum.
solution
78
2. Determine when
is a maximum if
Assume A is a positive definite matrix.
solution
l is the Lagrange multiplier.
This shows that is an eigenvector of A.
Thus is the eigenvector of A associated
with the largest eigenvalue, l.
79
Differentiation with respect to a matrix

• Let X denote a q p matrix. Let f (X) denote a
  function of the components of X then

80
Example

• Let X denote a p p matrix. Let f (X) ln X

Solution
Note Xij are cofactors
(i,j)th element of X-1
81
Example

• Let X and A denote p p matrices.

Let f (X) tr (AX)
Solution
82
Differentiation of a matrix of functions

• Let U (uij) denote a q p matrix of functions
  of x then

83

• Rules

84

• Proof

85

• Proof

86

• Proof

87
The Generalized Inverse of a matrix
88

• Recall
• B (denoted by A-1) is called the inverse of A if
• AB BA I

• A-1 does not exist for all matrices A
• A-1 exists only if A is a square matrix and A ?
  0
• If A-1 exists then the system of linear equations
  has a unique solution

89

• Definition
• B (denoted by A-) is called the generalized
  inverse (Moore Penrose inverse) of A if
• 1. ABA A
• 2. BAB B
• 3. (AB)' AB
• 4. (BA)' BA

Note A- is unique Proof Let B1 and B2
satisfying 1. ABiA A 2. BiABi Bi 3. (ABi)'
ABi 4. (BiA)' BiA
90
Hence B1 B1AB1 B1AB2AB1 B1 (AB2)'(AB1)
' B1B2'A'B1'A' B1B2'A' B1AB2 B1AB2AB2
(B1A)(B2A)B2 (B1A)'(B2A)'B2
A'B1'A'B2'B2 A'B2'B2 (B2A)'B2 B2AB2 B2
The general solution of a system of Equations
The general solution
91
Suppose a solution exists
Let
92
Calculation of the Moore-Penrose g-inverse
Let A be a pq matrix of rank q lt p,
Proof
thus
also
93
Let B be a pq matrix of rank p lt q,
Proof
thus
also
94
Let C be a pq matrix of rank k lt min(p,q),
then C AB where A is a pk matrix of rank k and
B is a kq matrix of rank k
Proof
is symmetric, as well as