Logistic Regression

1
Logistic Regression

• The usual regression model assumes that y is a
  fn(xs) e, where e has a normal distn
• This works well when y is SAT or cost or yield
• In logistic regression, y is a discrete variable
• Usually, y0 or 1 based on whether some event
  occurs or not
• In these cases, it doesnt make sense to have an
  error term that has a normal distn

2
Logistic Regression

• This turns out to mean that least squares doesnt
  work, either
• New principle Maximum Likelihood

3
Logistic Regression

• Suppose we have a random sample x1, x2, xn
• Prob distn of X depends on a parameter U
• For a given value of U, the likelihood of the
  sample is
• L(U) PROD Prob(xi U)

4
Logistic Regression

• Since the observations are independent, it makes
  sense to multiply probabilities
• We would want to find U that maximizes L(U)
• It is the value of the parameter for which our
  sample is most likely

5
Logistic Regression

• In many settings, the Maximum Likelihood
  Estimator (MLE) is the obvious one
• For a binomial, the MLE for p is the fraction of
  successes
• For a normal with known SD, the MLE for the mean
  is the avg

6
Logistic Regression

• Our plan is to use ML to estimate the parameters
  in our model for y
• odds p/(1-p)
• For p1/2, Odds1
• The lim as p-gt1 is infinity
• The lim as p-gt0 is 0

7
Logistic Regression

• Can fix the asymmetry if we take logs
• Log Odds ln(p/(1-p))
• LO(1/2)0
• LO is symmetric about p1/2
• We will let the LO be a linear fn of the xs

8
Logistic Regression

• Let B(x) b0b1x1 bpxp
• Ln(p/(1-p)) B(x)
• Pexp(B(x))/(1exp(B(x)))
• 1-P 1/(1exp(B(x)))

9
Logistic Regression

• Consider each case in our dataset
• If the event occurred then
• Pexp(B(x))/(1exp(B(x)))
• If the event didnt occur then use
• 1-P 1/(1exp(B(x)))

10
Logistic Regression

• The overall likelihood is made of both these
  products
• We wish to find the coefficients in B(x) to
  maximize the overall likelihood
• May be easier to maximize the log of the
  likelihood
• It will be the sum of the logs of the terms

11
Discrete Example

• Consider this example from Ch8 of MooreMcCabe,
  3rd Ed
• Survey of men and women about binge drinking

12
Discrete Example
13
Discrete Example

• Let X1 for men and X0 for women
• Men log(p/1-p)b0b1
• Women log(p/1-p)b0
• Estimate using the log-odds for each group
• B0b1log(1630/5550)-1.22522
• B0log(1684/8232)-1.5868
• B10.361639

14
Discrete Example

• Then b1LO(men)-LO(women)
• log(Odds(men)/Odds(women)
• log odds ratio
• So exp(b1) odds ratio
• B10.36
• Exp(b1) 1.43
• So odds for men are 1.43 greater than for women
• Often what is meant by 43 more likely
• Odds are 43 higher, not the probability is 43
  higher

15
Estimation in general

• If X is not discrete then we will have to use
  maximum likelihood to estimate parameters
• See logistic.m and logregr.m

16
Estimation in general

• Alternative
• b,dev,statsglmfit(x,y,'binomial','link','logit'
  )
• Dev2lk
• Stats.tb./sdcoeff
• Stats.pp-values (uncorrected)

17
Estimation in general

• Y can either be col of 1s and 0s
• For binge problem, Y can have 2 cols
• Y(,1) successes
• Y(,2) trials
• If X1 for Male

18
Estimation in general

• b
• -1.5869
• 0.3616
• gtgt stats.t
• -59.3326
• 9.3097
• So slope is 9.3 SD from 0
• So Indicator for Men is important
• Binge drinking depends on gender

19
Logistic Regression

• Test significance
• In ordinary regression, we use an F test
• Because we have the unknown SD of the es and
  taking the F ratio cancels the SD
• In logistic regression, we do not have an unknown
  SD to worry about

20
Logistic Regression

• General test
• -2 log-likelihood has a Chi-sq distn with dfp
  where p is the number of parameters being
  estimated from the data
• Called deviance in LR

21
Logistic Regression

• In ordinary regression, we use a partial F test
  to determine if we should add a variable
• In logistic regression, use the change in
  -2log-likelihood. This should have a Chi-sq with
  dfchange in df

22
Logistic Regression

• Example
• Low birth weight
• Lowbwt.xls
• xage wt smoke
• b
• 1.3682
• -0.0390
• -0.0121
• 0.6708
• stats.p
• 0.1773
• 0.2334
• 0.0479
• 0.0396

23
Logistic Regression

• So it would appear that Age is not important and
  Weight is
• Leave out age and calculate Dev2
• chiprob(1,dev2-dev1,99)
• 0.2267
• Which is approx stats.p

24
Logistic Regression

• Keep Age and leave out Wt
• chiprob(1,dev3-dev1,99)
• 0.0360
• Which is also approx stats.p

25
Logistic Regression

• In linear regr, when we compute F for the
  (overall) regression, we are actually computing
  partial F between just the intercept and the
  model with the Xs, as well
• In logistic, the overall model is again the
  difference between just the intercept and the
  model with the Xs as well

26
Logistic Regression

• Two ways to compute overall significance
• (1) logregra(,y)
• (2) glmfit with a col of 1s. This will generate
  an error, but will produce the correct output

27
Logistic Regression

• gtgt b,lklogregra(,y)lk0lk
• 117.34
• gtgt b,lklogregra(x,y)lk1lk
• 111.44
• gtgt a(lk0-lk1)2 Need twice likelihood
• 11.793
• gtgt chiprob(3,a,99)
• 0.0081283

28
Logistic Regression

• So, just like in linear regr, we can either use
  coeff/SD or the change in Dev
• But if we have several indicators, we must use
  change in Dev

29
Logistic Regression

• Consider Race
• Race1 for White, 2 for Black, 3 for Other
• (Omit first col so we are comparing to White)
• Change in Dev9.3260 for 2 indicator vars
• Chiprob0.0094, so Race matters

30
Logistic Regression

• Using Wt, Smoke, Race all matter
• Is the effect of Wt the same for all races?
• Consider interactions
• wtxrxprod(wt,ir)
• chiprob(2,dev4-dev5,99)
• 0.4728
• So interactions not important

31
Logistic Regression

• Is the effect of Smoke the same for all races?
• First be sure Race has an effect
• Just Age, Wt, Dev224.3407
• Age, Wt, Race, Dev215.0147
• PV (df2) is 0.0094
• Age, Wt, Race, Xprod Dev212.6114
• PV (df2) is 0.3007