James R' Stacks, Ph'D'

1
Introduction to Multiple Regression
James R. Stacks, Ph.D.
james_stacks_at_tamu-commerce.edu
The best way to have a good idea is to have lots
of ideas Linus Pauling
2
Standardized form of a regression equation with
three predictor variables
Zc b1Zp1 b2Zp2 b3Zp3
3
Predictor variables (standardized z scores)
Zc b1Zp1 b2Zp2 b3Zp3
4
Zc b1Zp1 b2Zp2 b3Zp3
Standardized regression coefficients
5
Zc b1Zp1 b2Zp2 b3Zp3
Predicted criterion score (zc ze)
6
Predictor variables (standardized z scores)
Zc b1Zp1 b2Zp2 b3Zp3
Predicted criterion score (zc ze)
Standardized regression coefficients
7
Zc b1Zp1 b2Zp2 b3Zp3
Predicted criterion score (zc ze)
8
Predicted criterion score (zc ze)
Zc b1Zp1 b2Zp2 b3Zp3
Recall that the predicted criterion score is the
is the actual criterion score minus the error
Zc b1Zp1 b2Zp2 b3Zp3 Ze
9
Recall that multiplication of an entire equation
by any value results in an equivalent
equation ybx is the same as yx bxx or
as yx bx2
10
The following demonstration of solving for
standardized regression coefficients is taken
largely from Maruyama, Geoffrey M.
(1998). Basics of structural equation modeling.
Thousand Oaks, CA Sage Publications, Inc.
11
Lets write three equivalent forms of the
previous multiple regression equation by
multiplying the original equation by each of the
three predictor variables
ZcZp1 b1Zp1Zp1 b2Zp2Zp1 b3Zp3Zp1 ZeZp1
ZcZp2 b1Zp1Zp2 b2Zp2Zp2 b3Zp3Zp2 ZeZp2
ZcZp3 b1Zp1Zp3 b2Zp2Zp3 b3Zp3Zp3 ZeZp3
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
12
Now notice all the zz cross products in the
equations. Recall that the expected (mean) cross
product is something we are familiar with. The
unbiased estimate of the cross product for paired
z values is E(cross product) S(zz)/(n-1)
, or , Pearson r !
ZcZp1 b1Zp1Zp1 b2Zp2Zp1 b3Zp3Zp1 ZeZp1
ZcZp2 b1Zp1Zp2 b2Zp2Zp2 b3Zp3Zp2 ZeZp2
ZcZp3 b1Zp1Zp3 b2Zp2Zp3 b3Zp3Zp3 ZeZp3
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
13
The Pearson product-moment correlation
coefficient (written as r for sample estimate, r
for parameter)
n

• S

r
Za Zb

n-1
i 1
Where za and zb are z scores for each person on
some measure a and some measure b, and n is the
number of people
14
So, I could just as easily write
rc p1 b1r p1 p1 b2 rp2 p1 b3 rp3 p1 re p1
rc p2 b1r p1 p2 b2 rp2 p2 b3 rp3 p2 re p2
rc p3 b1r p1 p3 b2 rp2 p3 b3 rp3 p3 re p3
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
15
Now, lets look at some interesting things about
the correlation coefficients we have substituted
rc p1 b1r p1 p1 b2 rp2 p1 b3 rp3 p1 re p1
rc p2 b1r p1 p2 b2 rp2 p2 b3 rp3 p2 re p2
rc p3 b1r p1 p3 b2 rp2 p3 b3 rp3 p3 re p3
Correlations of variables with themselves are
necessarily unity, So the red values are 1
In regression, error by definition is the
variance which does not correlate with any
variable, thus the blue values are necessarily 0
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
16
rc p1 b1(1) b2 rp2 p1 b3 rp3 p1
rc p2 b1r p1 p2 b2 (1) b3 rp3 p2
rc p3 b1r p1 p3 b2 rp2 p3 b3 (1)
The above system can be written in matrix form
rc p1
b1(1) b2 rp2 p1 b3 rp3 p1

rc p2
b1r p1 p2 b2 (1) b3 rp3 p2
rc p3
b1r p1 p3 b2 rp2 p3 b3 (1)
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
17
rc p1
b1(1) b2 rp2 p1 b3 rp3 p1

rc p2
b1r p1 p2 b2 (1) b3 rp3 p2
rc p3
b1r p1 p3 b2 rp2 p3 b3 (1)
Note that the matrix on the right side above is a
vector, and it is a product of a correlation
matrix of the predictor variables and a b vector.
rc p1
b1
(1) rp2 p1 rp3 p1

rc p2
r p1 p2 (1) rp3 p2
b2
rc p3
b3
r p1 p3 rp2 p3 (1)
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
18
rc p1
b1
1 rp2 p1 rp3 p1

rc p2
r p1 p2 1 rp3 p2
b2
rc p3
b3
r p1 p3 rp2 p3 1
The moral of this story is assuming all the
Pearson correlations among variables are known
(they are easily calculated), we can use the
equation above to solve for the b vector, which
is the standardized regression coefficients.
Zc b1Zp1 b2Zp2 b3Zp3
(Maruyama, Geoffrey M. (1998). Basics of
structural equation modeling. Thousand Oaks, CA
Sage Publications, Inc.)
19
rc p1
b1
1 rp2 p1 rp3 p1

rc p2
r p1 p2 1 rp3 p2
b2
rc p3
b3
r p1 p3 rp2 p3 1
This is a matrix equation which can be symbolized
as Riy RiiBi From algebra, such an equation
can obviously be solved for Bi by dividing both
sides by Rii, but there is no such thing as
division in matrix math
The matrix notation used here corresponds to your
text Tabachnik, Barbara G. Fidell, Linda S.
(2001). Using multivariate statistics., 4th
Edition. Needham Heights, MA Allyn Bacon
20
What is necessary to accomplish the same goal is
to multiply both sides of the equation by the
inverse of Rii, written as Rii-1. Rii-1 Riy
Rii-1Rii Bi therefore Rii-1Riy Bi If you have
studied the appendix assigned on matrix
algebra,you know that, while matrix
multiplication is quite simple, matrix inversion
is a real chore!
The matrix notation used here corresponds to your
text Tabachnik, Barbara G. Fidell, Linda S.
(2001). Using multivariate statistics., 4th
Edition. Needham Heights, MA Allyn Bacon
21
rc p1
b1
1 rp2 p1 rp3 p1

rc p2
r p1 p2 1 rp3 p2
b2
rc p3
b3
r p1 p3 rp2 p3 1
Riy Rii Bi
To get the solution we must find the inverse of
the green shaded matrix Rii in order to get
Rii-1 for the equation Rii-1 Riy Bi
The matrix notation used here corresponds to your
text Tabachnik, Barbara G. Fidell, Linda S.
(2001). Using multivariate statistics., 4th
Edition. Needham Heights, MA Allyn Bacon
22
(No Transcript)
23
(No Transcript)
24
The following method of inverting a matrix is
taken largely from Swokowski, Earl W. (1979)
Fundamentals of College Algebra. Boston, MA
Prindle, Weber Scmidt
25
The first step is to form a matrix which has the
same number of rows as the original correlation
matrix of predictors, but has twice as many
columns. The original predictor correlations are
placed in the left half, and an equal order
identity matrix is place in the right half
(Swokowski, Earl W. (1979) Fundamentals of
College Algebra. Boston, MA Prindle, Weber
Scmidt)
(Identity matrix)
(Predictor correlations)
26
Though a series of calculations called elementary
row transformations, the goal is to change all
the numbers in the matrix so that the identity
matrix is on the left, and a new matrix is on the
right
(Swokowski, Earl W. (1979) Fundamentals of
College Algebra. Boston, MA Prindle, Weber
Scmidt)
Inverse Matrix
Identity Matrix
27
Swokowski, Earl W. (1979) Fundamentals of College
Algebra. Boston, MA Prindle, Weber Scmidt

• MATRIX ROW TRANSFORMATION THEOREM
• Given a matrix of a system of linear equations,
  each of the
• following transformations results in a matrix of
  an equivalent
• system of linear equations
• Interchanging any two rows
• Multiplying all of the elements in a row by the
  same nonzero
• real number k.
• Adding to the elements in a row k times the
  corresponding
• elements of any other row, where k is any real
  number.

28
1st transformation a2j a2j (-.488) a1j
29
2nd transformation a3j a3j (-.354) a1j
30
3rd transformation a2j a2j . 1/.761856
31
4th transformation a3j a3j (-.199248) a2j
32
5th transformation a3j a3j . 1/.822574723
33
6th transformation a1j a1j (-.488) a2j
34
7th transformation a1j a1j (-.226373488)
a3j
35
8th transformation a2j a2j (-.261529737)
a3j
36
Inverted matrix on right
INVERSION
Original matrix on left
37
beta vector
inverse of predictor correlations
predictor/criterion correlations
b1

b2
b3
Rii-1 Riy Bi
38
OUR CALCULATIONS
VALUES FROM SPSS
-.257
b1
b2
.873
.150
b3
The difference has to do with rounding error.
There are so many transformations in matrix math
that all computations must be carried out with
many, many significant figures, because the
errors accumulate. I only used what was visible
in my calculator. Good matrix software should
use much more precision. This is a relatively
brief equation to solve. Imagine the error that
can accumulate with hundreds of matrix
transformations. This is a very important point,
and one should always be certain the software is
using the appropriate degree of precision.,
39
The regression equation can then be written
Zc (-.255)Zp1 (.872)Zp2 (.149) Zp3