Generalized Linear Models May 16

1
Generalized Linear Models May 16 May 18

• Dingcai Cao
• d-cao_at_uchicago.edu

2
Multiple Regression
where y1, . . . , yn are n independent
observations on a response, and yi as a
realization of a random variable Yi. In
particular Yi has a normal distribution with mean
?i and variance ?2
3
Logistic Regression
y1, . . . , yn are n independent observations on
a response, and yi as a realization of a random
variable Yi. In particular Yi has a Bernoulli
distribution with probability of success to be
?i, which depends on X. Note a Bernoulli
distribution is a special case of binormial
distribution with N 1.
4
Loglinear Models
log(?ij)??iX?jY ?ijXY
5
Generalized Linear Models
What is in common for these models?
1) The left sides all have ???the expected value
of the response variable 2) The right sides are
all linear combinations of a few predictor
variables.
Can we describe the models in a more general form?
Yes!!! Generalized linear models
6
Generalized Linear Models
Generalized linear models (GLM) are a large class
of statistical models for relating responses to
linear combination of predictor variables.
Nelder, J. A. R. W. M. Wedderburn (1972).
Generalized Linear Models, Journal of the Royal
Statistical Society, Series A, 135, 370-384.
7
Generalized Linear Models
Generalized linear models (GLM) Dont get
confused with SAS procedure GLM (PROC
GLM) Since Nelder and Wedderburn published
their article in 1972, GLM has become the
standard name for generalized linear models.
However, PROC GLM is the SAS procedure for fixed
effects, normal errors linear models. PROC GLM
does not analyze generalized linear models at
all. Prior 1972, GLM was the accepted name for
the general linear model. Linear model
specialists agree that what was once known the
general linear model is not at all general by
contemporary standards. Therefore, the word
general is usually dropped. The name LM, for
linear model, in gainning acceptance for the
normal errors linear model. In summary, PROC GLM
analyzes LMs, PROC GENMOD analyzes GLMs.
8
GLM Components
Random component the probability distribution of
the response variable
Systematic component Linear combination of
predictor variables
Link function (g) the link between the random
and systematic components
9
GLM A Few More Details about the Theory
To use GLM, the distribution of y has to belong
to the exponential family, i. e.
10
GLM A Few More Details about the Theory
11
Goodness of Fits Deviance
Deviance -2LM-LS where LM is the maximum log
likelihood of the model of interest LS is the
maximum log likelihood for the most complex
model, which has a separated parameter at each
explanatory setting (saturated model). Deviance
has approximately a chi-square distribution with
df N-p Where N number of observations and p
number of parameters (including intercept).
Likelihood ratio test for model comparison
between M1 and M0 (M0 is a simpler model than
M1) Likelihood ratio -2L0-L1)2L0-LS--2L1-
LS Deviance0-Deviance1
12
Logistic Regression The Challenger Shuttle
O-Ring Data Example
Following the 1986 Challenger space shuttle
disaster, investigators focused on a suspected
association between O-ring failure and low
temperature at launch. Here the data documented
the presence or absence of O-ring thermal
distress in the 23 shuttle launches preceding the
Challenger mission.

Obs temp td no_td total
1 53
1 0 1
2 57 1 0
1
3 58 1 0 1
4 63
1 0 1
5 66 0 1
1
6 67 0 3 3
7 68
0 1 1
8 69 0 1
1
9 70 2 2 4
10
72 0 1 1
11 73 0
1 1
12 75 1 1 2
13
76 0 2 2
14 78 0
1 1
15 79 0 1 1

16 81 0 1 1
13
Logistic Regression The Challenger Shuttle
O-Ring Data Example
data Challenger input temp td no_td
totaltdno_td datalines 53 1 0 57 1 0 58 1
0 63 1 0 66 0 1 67 0 3 68 0 1 69 0 1 70 2 2 72 0
1 73 0 1 75 1 1 76 0 2 78 0 1 79 0 1 81 0 1
proc genmod dataChallenger model
td/totaltemp/distbinomial linklogit type1 run

Model Information
Data Set
WORK.CHALLENGER
Distribution
Binomial
Link Function Logit
Response
Variable (Events) td
Response Variable
(Trials) total
Observations Used
16
Number Of Events
7
Number Of Trials 23

Parameter Information
Parameter
Effect
Prm1 Intercept
Prm2
temp
14
Logistic Regression The Challenger Shuttle
O-Ring Data Example

Criteria For Assessing Goodness Of Fit
Criterion
DF Value Value/DF
Deviance
14 11.9974 0.8570
Scaled Deviance
14 11.9974 0.8570
Pearson Chi-Square 14
11.1303 0.7950
Scaled Pearson X2 14
11.1303 0.7950
Log Likelihood
-10.1576
Algorithm converged.

Analysis Of
Parameter Estimates
Standard Wald 95
Confidence Chi-
Parameter DF Estimate Error
Limits Square Pr ChiSq
Intercept 1 15.0429
7.3786 0.5810 29.5048 4.16
0.0415 temp 1
-0.2322 0.1082 -0.4443 -0.0200
4.60 0.0320 Scale
0 1.0000 0.0000 1.0000
1.0000
LR Statistics
For Type 1 Analysis

Chi- Source
Deviance DF Square Pr
ChiSq
Intercept 19.9494

temp 11.9974 1 7.95
0.0048
15
Logistic Regression The Challenger Shuttle
O-Ring Data Example

Convergence criterion (GCONV1E-8) satisfied.

Model Fit Statistics

Intercept
Intercept
and
Criterion Only Covariates
AIC
30.267 24.315
SC
31.403 26.586
-2 Log L 28.267
20.315
Testing Global Null Hypothesis BETA0
Test
Chi-Square DF Pr ChiSq
Likelihood Ratio
7.9520 1 0.0048
Score
7.2312 1 0.0072
Wald
4.6008 1 0.0320
Analysis of Maximum
Likelihood Estimates
Standard
Wald
Parameter DF Estimate Error
Chi-Square Pr ChiSq
Intercept 1 15.0429 7.3786
4.1563 0.0415
temp 1 -0.2322 0.1082
4.6008 0.0320
proc logistic dataChallenger model
td/totaltemp run
16
Logistic Regression The Challenger Shuttle
O-Ring Data Example
Using Genmod to get predicted probability
proc genmod dataChallenger model
td/totaltemp/distbinomial linklogit type1
estimate 'logit at 50 deg' intercept 1 temp 50
estimate 'logit at 60 deg' intercept 1 temp 60
estimate 'logit at 64.7 deg' intercept 1 temp
64.7 estimate 'logit at 64.8 deg' intercept 1
temp 64.8 estimate 'logit at 70 deg' intercept
1 temp 70 estimate 'logit at 80 deg' intercept
1 temp 80 ods output estimateslogit run

Contrast Estimate Results
Standard
Chi-
Label Estimate Error
Alpha Confidence Limits Square Pr
ChiSq logit at 50 deg
3.4348 2.0232 0.05 -0.5307
7.4002 2.88 0.0896
logit at 60 deg 1.1131 1.0259
0.05 -0.8975 3.1238 1.18
0.2779 logit at 64.7 deg
0.0220 0.6576 0.05 -1.2669
1.3109 0.00 0.9733
logit at 64.8 deg -0.0012 0.6518
0.05 -1.2788 1.2764 0.00
0.9985 logit at 70 deg
-1.2085 0.5953 0.05 -2.3752
-0.0418 4.12 0.0423
logit at 80 deg -3.5301 1.4140
0.05 -6.3014 -0.7588 6.23
0.0125
17
Logistic Regression The Challenger Shuttle
O-Ring Data Example
Alternative analysis using 0-1 Data
data O_Ring input flt temp td datalines 1 66
0 2 70 1 3 69 0 4 68 0 5 67 0 6 72 0 7 73
0 8 70 0 9 57 1 10 63 1 11 70 1 12 78 0 13 67
0 14 53 1 15 67 0 16 75 0 17 70 0 18 81 0 19 76
0 20 79 0 21 75 1 22 76 0 23 58 1
proc freq tables temptd run proc genmod
dataO_Ring model tdtemp /distbinomial
linklogit type1 estimate 'logit at 50 deg'
intercept 1 temp 50 estimate 'logit at 60 deg'
intercept 1 temp 60 estimate 'logit at 64.7
deg' intercept 1 temp 64.7 estimate 'logit at
64.8 deg' intercept 1 temp 64.8 estimate 'logit
at 70 deg' intercept 1 temp 70 estimate 'logit
at 80 deg' intercept 1 temp 80 ods output
estimateslogit run
18

Model Information
Data Set
WORK.O_RING
Distribution Binomial
Link
Function Logit
Dependent Variable
td
Observations Used 23

Response Profile
Ordered
Total
Value td Frequency
1
1 7
2 0 16

Parameter Information
Parameter
Effect
Prm1 Intercept
Prm2
temp
Criteria For Assessing Goodness Of
Fit
Criterion DF Value
Value/DF
Deviance 21 20.3152
0.9674
Scaled Deviance 21 20.3152
0.9674
Pearson Chi-Square 21 23.1691
1.1033
Scaled Pearson X2 21 23.1691
1.1033
Log Likelihood -10.1576

Alternative analysis using 0-1 Data
19
Alternative analysis using 0-1 Data
Analysis Of Parameter
Estimates
Standard Wald 95 Confidence
Chi- Parameter DF
Estimate Error Limits
Square Pr ChiSq
Intercept 1 15.0429 7.3786
0.5810 29.5048 4.16 0.0415
temp 1 -0.2322
0.1082 -0.4443 -0.0200 4.60
0.0320 Scale 0
1.0000 0.0000 1.0000 1.0000

LR Statistics For Type 1
Analysis
Chi-
Source
Deviance DF Square Pr ChiSq
Intercept
28.2672
temp
20.3152 1 7.95 0.0048

Contrast Estimate Results
Standard
Chi-
Label Estimate Error
Alpha Confidence Limits Square Pr
ChiSq logit at 50 deg
3.4348 2.0232 0.05 -0.5307
7.4002 2.88 0.0896
logit at 60 deg 1.1131 1.0259
0.05 -0.8975 3.1238 1.18
0.2779 logit at 64.7 deg
0.0220 0.6576 0.05 -1.2669
1.3109 0.00 0.9733
logit at 64.8 deg -0.0012 0.6518
0.05 -1.2788 1.2764 0.00
0.9985 logit at 70 deg
-1.2085 0.5953 0.05 -2.3752
-0.0418 4.12 0.0423
logit at 80 deg -3.5301 1.4140
0.05 -6.3014 -0.7588 6.23
0.0125
20
Probit Regression The Challenger Shuttle O-Ring
Data Example An Alternative Link to
Success/Failure Data
Assumption of Probit regression z ?X has a
standard normal distribution If z is below a
certain level, you observe a success. Otherwise,
you observe a failure. Therefore, the probability
of success is the area under the normal curve
below the certain level.
21
Probit Regression The Challenger Shuttle O-Ring
Data Example An Alternative Link to
Success/Failure Data

Model Information
Data Set
WORK.CHALLENGER
Distribution
Binomial
Link Function Probit
Response
Variable (Events) td
Response Variable
(Trials) total
Observations Used
16
Number Of Events
7
Number Of Trials 23

Parameter Information
Parameter
Effect
Prm1 Intercept
Prm2
temp
proc genmod dataChallenger model
td/totaltemp/linkprobit type1 estimate
'probit at 50 deg' intercept 1 temp 50 estimate
'probit at 60 deg' intercept 1 temp 60 estimate
'probit at 64.7 deg' intercept 1 temp 64.7
estimate 'probit at 64.8 deg' intercept 1 temp
64.8 estimate 'probit at 70 deg' intercept 1
temp 70 estimate 'probit at 80 deg' intercept 1
temp 80 ods output estimatesprobit run
22
Probit Regression The Challenger Shuttle O-Ring
Data Example An Alternative Link to
Success/Failure Data

Criteria For Assessing Goodness Of Fit
Criterion
DF Value Value/DF
Deviance
14 12.0600 0.8614
Scaled Deviance
14 12.0600 0.8614
Pearson Chi-Square 14
10.9763 0.7840
Scaled Pearson X2 14
10.9763 0.7840
Log Likelihood
-10.1889
Analysis Of Parameter
Estimates
Standard Wald 95 Confidence
Chi- Parameter DF
Estimate Error Limits
Square Pr ChiSq
Intercept 1 8.7750 4.0286
0.8790 16.6709 4.74 0.0294
temp 1 -0.1351
0.0584 -0.2495 -0.0206 5.35
0.0207 Scale 0
1.0000 0.0000 1.0000 1.0000
23
Probit Regression The Challenger Shuttle O-Ring
Data Example An Alternative Link to
Success/Failure Data
LR
Statistics For Type 1 Analysis

Chi-
Source Deviance DF Square
Pr ChiSq
Intercept 19.9494

temp 12.0600 1 7.89
0.0050
Contrast Estimate Results

Standard
Chi- Label
Estimate Error Alpha Confidence
Limits Square Pr ChiSq
probit at 50 deg 2.0201 1.1413
0.05 -0.2167 4.2570 3.13
0.0767 probit at 60 deg
0.6692 0.6024 0.05 -0.5115
1.8498 1.23 0.2666
probit at 64.7 deg 0.0342 0.3960
0.05 -0.7420 0.8104 0.01
0.9312 probit at 64.8 deg
0.0207 0.3925 0.05 -0.7487
0.7901 0.00 0.9579
probit at 70 deg -0.6818 0.3244
0.05 -1.3175 -0.0461 4.42
0.0356 probit at 80 deg
-2.0328 0.7277 0.05 -3.4590
-0.6066 7.80 0.0052
24
Probit Regression The Challenger Shuttle O-Ring
Data Example An Alternative Link to
Success/Failure Data

Standard
Chi- Label
Estimate Error Alpha Confidence
Limits Square Pr ChiSq
probit at 50 deg 2.0201 1.1413
0.05 -0.2167 4.2570 3.13
0.0767 probit at 60 deg
0.6692 0.6024 0.05 -0.5115
1.8498 1.23 0.2666
probit at 64.7 deg 0.0342 0.3960
0.05 -0.7420 0.8104 0.01
0.9312 probit at 64.8 deg
0.0207 0.3925 0.05 -0.7487
0.7901 0.00 0.9579
probit at 70 deg -0.6818 0.3244
0.05 -1.3175 -0.0461 4.42
0.0356 probit at 80 deg
-2.0328 0.7277 0.05 -3.4590
-0.6066 7.80 0.0052
logit at 50 deg 3.4348 2.0232
0.05 -0.5307 7.4002 2.88
0.0896 logit at 60 deg
1.1131 1.0259 0.05 -0.8975
3.1238 1.18 0.2779
logit at 64.7 deg 0.0220 0.6576
0.05 -1.2669 1.3109 0.00
0.9733 logit at 64.8 deg
-0.0012 0.6518 0.05 -1.2788
1.2764 0.00 0.9985
logit at 70 deg -1.2085 0.5953
0.05 -2.3752 -0.0418 4.12
0.0423 logit at 80 deg
-3.5301 1.4140 0.05 -6.3014
-0.7588 6.23 0.0125
Overall, logit link and probit link fit the data
similarly well. However, in some situations with
extreme probability (close to 0 or 1), logit link
is better.
25
Poisson Regression for Count Data
Insect control data
Obs
block trt ctl_trt a b count
1 1
0 0 0 0 16
2 1 1
1 1 1 6
3 1 2 1
1 2 2
4 1 3 1 1 3
4
5 1 4 1 2 1 5
6
1 5 1 2 2 3
7 1
6 1 2 3 1
8 1 7
1 3 1 1
9 1 8 1
3 2 3
10 1 9 1 3 3
1
11 2 0 0 0 0 25
12
2 1 1 1 1 9
13 2
2 1 1 2 6
14 2 3
1 1 3 3
15 2 4 1
2 1 3
16 2 5 1 2 2
4
17 2 6 1 2 3 5
18
2 7 1 3 1 2
19 2
8 1 3 2 2
20 2 9
1 3 3 0
4 blocks Each block has 10 treatments Control
(1 level), trt A (3 levels), trt B (3
levels) Response is insect count
26
Poisson Regression for Count Data
Insect control data
proc genmod datacounts class BLOCK CTL_TRT a
b model countBLOCK CTL_TRT a b ab/distnegbin
type1 type3 wald title 'uncorrected Poisson
model' run

The GENMOD Procedure
Model
Information
Data Set WORK.COUNTS

Distribution Negative Binomial
Link Function
Log
Dependent Variable
count
Observations Used
40
Class Level Information
Class
Levels Values
block 4 1 2 3 4

ctl_trt 2 0 1
a
4 0 1 2 3
b 4 0 1 2 3
27
Poisson Regression for Count Data
Insect control data

Criteria For Assessing Goodness Of Fit
Criterion
DF Value Value/DF
Deviance
27 36.7953 1.3628
Scaled Deviance
27 36.7953 1.3628
Pearson Chi-Square 27
33.8797 1.2548
Scaled Pearson X2 27
33.8797 1.2548
Log Likelihood
212.7270
28
Poisson Regression for Count Data
Insect control data
Analysis Of
Parameter Estimates
Standard Wald
95 Confidence Chi-
Parameter DF Estimate
Error Limits Square Pr
ChiSq Intercept 1
1.2085 0.4025 0.4196 1.9975
9.01 0.0027 block
1 1 -0.4756 0.3164
-1.0958 0.1446 2.26 0.1328
block 2 1 -0.2198
0.3063 -0.8200 0.3805 0.51
0.4730 block 3
1 0.0210 0.3076 -0.5819
0.6239 0.00 0.9455
block 4 0 0.0000
0.0000 0.0000 0.0000 .
. ctl_trt 0 1
1.6422 0.4821 0.6974 2.5871
11.61 0.0007 ctl_trt
1 0 0.0000 0.0000
0.0000 0.0000 . .
a 0 0 0.0000
0.0000 0.0000 0.0000 .
. a 1
1 0.3946 0.5160 -0.6169
1.4060 0.58 0.4445 a
2 1 0.6847 0.5060
-0.3071 1.6764 1.83 0.1760
a 3 0
0.0000 0.0000 0.0000 0.0000
. . b 0
0 0.0000 0.0000 0.0000
0.0000 . .
b 1 1 -0.3728
0.5762 -1.5021 0.7565 0.42
0.5176 b 2 1
0.0472 0.5390 -1.0092 1.1037
0.01 0.9302 b
3 0 0.0000 0.0000
0.0000 0.0000 . .
ab 0 0 0 0.0000
0.0000 0.0000 0.0000 .
. ab 1 1
1 1.1901 0.7372 -0.2547
2.6350 2.61 0.1064
ab 1 2 1 0.3368
0.7138 -1.0622 1.7358 0.22
0.6371 ab 1 3 0
0.0000 0.0000 0.0000 0.0000
. . ab
2 1 1 -0.1498 0.7610
-1.6413 1.3417 0.04 0.8439
ab 2 2 1 -0.4022
0.7210 -1.8153 1.0109 0.31
0.5770 ab 2 3
0 0.0000 0.0000 0.0000
0.0000 . .
ab 3 1 0 0.0000
0.0000 0.0000 0.0000 .
. ab 3 2 0
0.0000 0.0000 0.0000 0.0000
. . ab
3 3 0 0.0000 0.0000
0.0000 0.0000 . .
Dispersion 1 0.2383
0.0899 0.1137 0.4992