Metaanalysis

1
Meta-analysis

• Funded through the ESRCs Researcher Development
  Initiative

Session 2.4 3-level meta-analyses
Department of Education, University of Oxford
2
Steps in a meta-analysis
Session 2.4 3-level meta-analyses
3
Odds Ratio effect sizes
4
Effect size calculation

• The odds-ratio is based on a 2 by 2 contingency
  table
• The Odds-Ratio is the odds of success in the
  treatment group relative to the odds of success
  in the control group (in the present application
  males and females)

5
Effect size calculation
XLS
6
Gender bias in peer reviews
7
Gender Differences in Peer Review Bornmann,
L. (2007). Bias cut. Women, it seems, often get a
raw deal in scienceSo how can discrimination be
tackled? Nature, 445(7127), 566. Bornmann,
L., Mutz, R. Daniel, H. D. (2007). Gender
differences in grant peer review A
meta-analysis. Journal of Informetrics, 1,
226238.
Abstract Narrative reviews of peer review
research have concluded that there is negligible
evidence of gender bias in the awarding of grants
based on peer review. Here, we report the
findings of a meta-analysis of 21 studies
providing, to the contrary, evidence of robust
gender differences in grant award procedures.
Even though the estimates of the gender effect
vary substantially from study to study, the model
estimation shows that all in all, among grant
applicants men have statistically significant
greater odds of receiving grants than women by
about 7
8
Gender Differences in Peer Review
Bornmann et al. conducted a multilevel to
meta-analysis based peer reviews (grant
applications fellowship applications) 66
effect sizes from 21 studies and a total of
353,725 applications. They found a statistically
significant but small effect in favour of men (an
effective odds-ratio of 1.07). However, there was
systematic variation in the effect sizes beyond
random sampling error, suggesting that their
results were not generalizable. Typically the
next step would be to consider moderators type
of application (grants vs. pre- post-doctoral
fellowships), discipline, or country. However,
they noted that Unfortunately, the inclusion
of one or more of these characteristics into the
calculation of the meta-analysis resulted in
models that did not converge in the estimation
process. This finding indicated that the model
estimation became too complex by considering
specific interaction effects or the included
characteristics had no influence on the outcome,
respectively.
9
Gender Differences in Peer Review (Meta-analysis
data from Bornmann, Mutza, Daniel,, 2007
Are women disadvantaged in Peer Reviews? Based on
66 outcomes from 21 studies, we evaluate whether
there are systematic gender differences in
success. Important moderator variables include
type of peer review (grants, fellowships),
discipline, country, and year.
9
10
Gender Differences in Peer Review
10
11
Peer Review Mean Unwted Study Level
12
Peer Review Mean Wted by Ntot
13
Preliminary Box Plots Total Sample
Weighted
Unweighted
Potential Outliers
No Gender Difference
75th tile
Median Effect Size slightly in favour of women
Median Effect Size slightly in favour of men
25th tile
Potential Outliers
14
Preliminary Box Plots Type
No Gender Difference
15
Preliminary Box Plots Country
No Gender Difference
16
Preliminary Box Plots Discipline
No Gender Difference
17
Preliminary Box Plots Year
No Gender Difference
18
Random effects models
19
Random effects assumptions

• Is only a sample of studies from the entire
  population of studies to be considered. As a
  result, do want to generalise to other studies
  not included in the sample (e.g., future
  studies).
• Variability between effect sizes is due to
  sampling error plus variability in the population
  of effects.
• In contrast to fixed effects models, there are 2
  sources of variance
• Effect sizes are independent.

20
Random effects
Where dj is the observed effect size in study
j d is the mean true population effect size uj
is the deviation of the true study effect size
from the mean true effect size and ej is the
residual due to sampling variance in study j
21
Conducting random effects meta-analysis

• Like the fixed effects model, there are 2 general
  ways of conducting a random effects
  meta-analysis ANOVA multiple regression
• The analogue to the ANOVA homogeneity analysis is
  appropriate for categorical variables
• Looks for systematic differences between groups
  of responses within a variable
• Multiple regression homogeneity analysis is more
  appropriate for continuous variables and/or when
  there are multiple variables to be analysed
• Tests the ability of groups within each variable
  to predict the effect size
• Can include categorical variables in multiple
  regression as dummy variables

22
SPSS Commands
23
MeanES MACRO
Conclusions Small effect size based on both
Fixed Random models. Slightly in favour of
females for Fixed effects, slightly in favour of
males for random effects Significant
study-to-study variation so random effects and
search for moderators appropriate.
24
MetaF MACRO (ANOVA)Type (0grants, 1fellowships)
25
MetaF MACRO (ANOVA)Discipline 1Phys 2Biomed
3SocSc 4Mult 5human
26
MetaF MACRO (ANOVA) Country 1Australia
2Canada 3Germany 4Europe 5Netherlands
6Sweden 7UK 8USA
Sweden
Conclusions BIG difference in favour of men in
Sweden smaller differences in favour of men in
Germany and Europe NS differences for other
countries.
27
Multilevel Models
28
Website Address to get MLWIN
Harvey Goldstein developed the MLWIN statistical
package used here and has made many contributions
to multilevel modeling, including meta-analysis.
29
Setting Up Meta-analysis
2

• Click on the equation
• make logOR the y variable
• indicate a three level model with L3study,
  L2id, L3LogOR
• Click done button

3
4
30
Setting Up Meta-analysis

• Click Cons in the equation
• Tick Fixed Parameter (study) i(d) but not
  logOR
• Click the done button

31
Setting Up Meta-analysis

• Now click add term button
• This will bring up the X-Variable select SE
  (the standard error computed earlier)
• Tick only the logOR box
• Click done

32
Setting Up Meta-analysis
Now we want to constrain the variance at level 1
to be fixed at 1.0. Under model select
constrain parameters will bring up parameter
constraint window
33
Setting Up Meta-analysis
In the parameter constraint window 1. Click the
random button 2.Change logOR SE/SE to 1 3.
Change to equal to 1 4. store the
constraints in the first empty column (here
C27) 5. Click the attach random constraints
button 6. Close Parameter Constraint Window
34
null model with no predictors
Conclusion The mean effect size (-.101/.040) is
significant. The chi-square (389.88) is signif
there is study-to-study variation. explore
moderator variables
-gtpred c50-gtcalc c51(('logOR'-c50)/'se')2-gtsum
c51 to b1 389.88 -gtcprob b1 65 5.6052e-045
After Closing the parameter constraint window
(last slide) Click on start button in
equation window (may have to click estimates
button to get values). Compute chi-square value
in command interface window
35
Add Type (0grant, 1fellow)
Conclusion The effect of type (-.196/.052) is
highly significant The mean effect size
(-.007/.034) NS for Type grant
(intercept) chi-sq (171.34) signif remaining
study-to-study variation.
36
Add DISC 1Phys 2Biomed 3SocSc 4Mult
5human
-gtpred c50-gtcalc c51 (('logor' -
c50)/'se')2-gtsum c51 to b1 188.59 -gtcprob b1
61 .1875e-014
Conclusion The effect of DISC is highly
significant (change in chi-sq 389.88 -188.59
200.29 (df 4). Men signif more successful than
women in SocSci (relative to multidis, the
reference category that is NS.
37
Add CNTRY 1Australia 2Canada 3Germany
4Europe 5Netherlands 6Sweden 7UK 8USA
gtpred c50-gtcalc c51 (('logor'-c50)/'se')2-gtsum
c51 to b1 158.76 -gtcprob b1 58 1.7144e-011
Conclusion The effect of CNTRY is highly
significant (change in chi-sq 389.88 -158.76
189.59 (df 7). Men signif more successful is
Swenden (but note large SE) and Germany relative
to US (reference category which is NS).
38
Add Year
-gtpred c50-gtcalc c51 (('logor' -
c50)/'se')2-gtsum c51 to b1 344.02 -gtcprob b1
64 6.6568e-038
Conclusion The Linear effect of YEAR is NS.
Notice that I changed the intercept to be 2000
(rather than 0 which is completely out of the
range.
39
Add Type DISC 1Phys 2Biomed 3SocSc
4Mult 5human
Note that solution is technically improper (study
level constrained to be non-negative) -gtpred
c50-gtcalc c51 (('logor' - c50)/'se')2-gtsum
c51 to b1 105.47 -gtcprob b1 60 0.00026315
Conclusion General pattern of results for each
variable considered separately still evident.
Reference category (Type grants, Disc Multi)
still NS. Results should be interpreted
cautiously because improper solution.
40
Add Type x DISC Interact 1Phys 2Biomed
3SocSc 4Mult 5human
Note that solution is technically improper (study
level constrained to be non-negative) -gtpred
c50-gtcalc c51 (('logor' - c50)/'se')2-gtsum
c51 to b1 103.80 -gtcprob b1 56 0.00010859
Conclusion The change in chi-sq is NS,
suggesting that there is no interaction. Results
should be interpreted cautiously because improper
solution.
41
Add Type CNTRY 1Australia 2Canada
3Germany 4Europe 5Netherlands 6Sweden 7UK
8USA
Conclusion General pattern of results similar.
Men signif more successful is Sweden (but note
large SE) and Germany relative to reference
category (US Grants).
42
Type x CNTRY Interaction 1Australia 2Canada
3Germany 4Europe 5Netherlands 6Sweden 7UK
8USA
Conclusion The change in chi-sq is NS,
suggesting that there is no interaction. Results
should be interpreted cautiously because improper
solution.
43
Main Effects of Type, Disc Country
Conclusion When all main effects are included,
Type effect nearly unaffected. However, none of
the disc effects are significant, although the
Sweden and (marginally) Germany are still
significant. Results should be interpreted
cautiously because improper solution.
44
Graphs
45
Graphs Caterpillar Plots
Caterpillar plot based on L1 residuals. Go to
the model menu and select residuals option.
This will bring up the settings window. Set SD
(comparative) to 1.96 3. Set level to
1logOR 4. click the Calc button 5. click on
the plot button to bring up the next window. In
the plot window select residual /1 1.96SD x
rank. This brings up the original graph. Clicking
on the graph bring up a window to modify the
graph (a bit)
46
Conclusion
47
Summary

• The mean effect size was very small, but
  significantly in favour of men. However, the
  results did not generalise across studies (there
  was study-to-study variation).
• The effect size was significantly moderated by
  the type it was almost exactly 0 for grants and
  in favour of men for fellowship applications.
  This difference was not moderated or mediated by
  other moderators.
• There appeared to be some discipline effects
  (bias in favour of men in social sciences) and
  country effects (large bias in favour of men for
  Sweden). However, when all main effects
  included, discipline effects disappeared.
• For Grant Proposals there was no evidence of any
  effect of gender on outcome.

48
Software

• Purpose-built
• Comprehensive Meta-analysis (commercial)
• Schwarzer (free, http//userpage.fu-berlin.de/hea
  lth/meta_e.htm)
• Extensions to standard statistics packages
• SPSS, Stata and SAS macros, downloadable from
  http//mason.gmu.edu/dwilsonb/ma.html
• Stata add-ons, downloadable from
  http//www.stata.com/support/faqs/stat/meta.html
• HLM V-known routine
• MLwiN
• MPlus

49
Key references

• Bornmann, L. (2007). Bias cut. Women, it seems,
  often get a raw deal in scienceSo how can
  discrimination be tackled? Nature, 445(7127),
  566.
• Bornmann, L., Mutz, R. Daniel, H. D. (2007).
  Gender differences in grant peer review A
  meta-analysis. Journal of Informetrics, 1,
  226238.
• Cooper, H., Hedges, L. V. (Eds.) (1994). The
  handbook of research synthesis (pp. 521529). New
  York Russell Sage Foundation.
• Hox, J. (2003). Applied multilevel analysis.
  Amsterdam TT Publishers.
• Hunter, J. E., Schmidt, F. L. (1990). Methods
  of meta-analysis Correcting error and bias in
  research findings. Newbury Park Sage
  Publications.
• Lipsey, M. W., Wilson, D. B. (2001). Practical
  meta-analysis. Thousand Oaks, CA Sage
  Publications.