Metaanalysis

1
Meta-analysis

• Funded through the ESRCs Researcher Development
  Initiative

Session 3.3 3.4 Teacher Expectancy Example
Department of Education, University of Oxford
2
Steps in a meta-analysis
Session 3.3 3.4 Teacher Expectancy Example
3
Teacher Expectancy Effects on IQ

• (Meta-analysis data from Raudenbush Bryk, 2002)

4
Research question

• Do teacher expectations influence student IQs?
• Teachers led to have high expectations of
  experimental (through bogus feedback) but not
  control students.
• The focus is on the effect of how long teachers
  knew students prior to the experimental
  intervention.

5
Teacher Expectancy Effects on IQ (Meta-analysis
data from Radudenbush Bryk, 2002
Do teacher expectations influence student IQs?
Teachers led to have high expectations of
experimental (through bogus feedback) but not
control students. Focus here is on the effect of
how long teachers knew students prior to the
experimental intervention.
6
Teacher Expectancy Effects on IQ (Meta-analysis
data from Radudenbush Bryk, 2002
6
7
Preliminary box plots
8
Fixed and random effects

• Mean effect size and homogeneity analyses

9
SPSS Commands
10
MeanES MACRO
2523 0 ----------------------------------------
---------------------- 2524 0 ' Macro for
SPSS/Win Version 6.1 or Higher 2525 0 '
Written by David B. Wilson (dwilson_at_crim.umd.edu)
2526 0 ' Meta-Analyzes Any Type of Effect
Size 2527 0 ' To use, initialize macro with
the include statement 2529 0 ' INCLUDE
"drivepathMeanES.SPS" . 2530 0 ' Syntax
for macro 2532 0 ' MeanES ESvarname
/Wvarname /PRINToption . 2534 0 ' E.g.,
MeanES ES D /W IVWEIGHT . 2535 0 ' In this
example, D is the name of the effect size
variable 2536 0 ' and IVWEIGHT is the name of
the inverse variance weight 2537 0 ' variable.
Replace D and INVWEIGHT with the
appropriate 2539 0 ' variable names for your
data set. 2540 0 ' /PRINT has the options
"EXP" and "IVZR". The former 2541 0 ' prints
the exponent of the results (odds-ratios)
and 2542 0 ' the latter prints the inverse Zr
transform of the 2543 0 ' results. If the
/PRINT statement is ommitted, the 2545 0 '
results are printed in their raw form.
11
MeanES MACRO
12
MeanES MACRO
Conclusions Small (NS) effect size based on both
Fixed Random models. Significant unexplained
variance suggesting non-generalisability of
effects across studies and the need for a random
effects model.
13
Fixed and random effects

• Analogue to the ANOVA analyses

14
MetaF MACRO (ANOVA)based on categorical weeks
Conclusions Large effect of weeks (20.4/35.8
57 var expl) Total Residual, variance
component residual by group all NS ES
significant for 1st two groups, NS last two groups
15
Fixed and random effects

• Regression analyses

16
MetaReg MACRO (Regression)based of categorical
weeks
Conclusions Large effect of weeks (54 var
expl) Constant term highly significant (at
intercept 0) Residual variance NS (variance
component 0)
17
MetaReg MACRO (Regression)based on uncategorical
weeks
Conclusions Large effect of weeks (21 var
expl) Constant term highly significant (at
intercept 0) Residual var NS Does not do as
well as categorised weeks
18
Fixed and random effects

• Variations of the previous analyses

19
MetaF MACRO (ANOVA)based of Blind vs. Aware Test
Administrators
Conclusions Small, NS effect Resid var
marginally significant for Aware not blind
20
MetaF MACRO (ANOVA)based of Group vs. Individual
IQ Tests
Conclusions Small, marginally significant
effect (4.1/34.312 var expl) ES NS for Group
but signi for individual Resid var for group
signif but individual NS All effects very small
21
MetaReg MACRO (Regression)based of Group vs.
Individual IQ Tests
Conclusions Small, marginally significant
effect (12 var expl) Constant term (Group) NS
Resid var signif (but variance component NS)
22
MetaReg MACRO (Regression)based of (Group vs.
Individual IQ Tests) (categorised weeks)
Conclusions Effect of test type no longer signif
when weeks included. Effect of weeks nearly
unaffected. Note can only look at multiple
variables with regression.
23
Multilevel models
24
Website Address to get MLwiN
Harvey Goldstein developed the MLwiN statistical
package used here and has made many contributions
to multilevel modeling, including meta-analysis.
25
Always a bit dangerous to say some one person
invented a new approach. However, fair to say
that Stephen Raudenbush at least popularised the
multilevel approach to meta-analysis with the
meta-analysis of the teacher-expectancy data
considered here. Raudenbush, S.W. and Bryk, A.S.
(2002).Hierarchical Linear Models (Second
Edition).Thousand Oaks Sage Publications, 482
pp. Raudenbush, S.W. (1984). Magnitude of teacher
expectancy effects on Pupil IQ as a function of
the credibility of expectancy induction A
synthesis of findings from 18 experiments.Journal
of Educational Psychology, 76, 1, 85-97.
26
Getting Data Into MLwiN
In an empty MLwiN file, puts the xx input
variables into first xx columns. (can also add
new data to existing files). Check to see that
data is correct and click on paste button
27
Getting Data Into MLwiN
Check the MLwiN names file to see that data
looks ok (e.g., missing values min max
values).
28
Setting Up Meta-analysis
MLwiN will open an empty equation that you have
to construct.
Click on the y to bring up this screen. select
d (the effect size) as the dependent
variable Select 2 for N of levels select ID
for Level 1 select d for Level 2
29
Setting Up Meta-analysis
3

• Click Add Term Button (bottom equations window)
• Select cons (variable 1 for all cases)
• Click the done button

30
Setting Up Meta-analysis

• Click Cons in the equation
• Tick Fixed Parameter j(id) but not i(d)
• Click the done button

31
Setting Up Meta-analysis

• Now click add term button
• This will bring up the X-Variable select SE
  (the standard error computed earlier)
• Tick only the i(d) box
• Click done

32
Setting Up Meta-analysis
Now we want to constrain the variance at level 1
to be fixed at 1.0. Under model select
constrain parameters will bring up parameter
constraint window
33
Setting Up Meta-analysis

• In the parameter constraint window
• Click the random button
• Change d SE/SE to 1
• Change to equal to 1

34
Setting Up Meta-analysis

• store the constraints in the first empty column
  (C19)
• Click the attach random constraints button.
• Close the Parameter Constraint Window

35
null model with no predictors
Conclusion The mean effect size (.078) is not
significant. The chi-square is significant there
is study-to-study variation. Reasonable to
explore moderator variables
After Closing the parameter constraint window
(last slide) Click on start button in
equation window (may have to click estimates
button to get values). Compute chi-square value
in command interface window
36
Add raw weeks variable
Conclusion The effect of weeks (-.013/.005) is
significant The mean effect size (.162/.055)
signif (when weeks 0). chi-sq signif some
remaining study-to-study variation.
37
WKCAT 4-category weeks
Conclusion categorized weeks does best of of
(chi-sq 16.568)
38
Aware vs. Blind Administration
For a categorical variable, you choose a
reference (left out category. Default is the
1st category
gtpred c50 -gtcalc c51 (('d' - c50)/'se')2
-gtsum c51 b1 35.608 -gtcprob b1 17 0.0051714
Conclusion Main Effect of Aware vs. Blind is NS
39
Aware vs. Blind Administration
For a categorical variable, you choose a
reference (left out category. Default is the
1st category
gtpred c50 -gtcalc c51 (('d' - c50)/'se')2
-gtsum c51 b1 16.445 -gtcprob b1 17 0.49254
Conclusion Effect for Blind vs. Aware reduced by
controlling for wkcat but was already
nonsignificant
40
Individual vs. Group Tests
Conclusion Effect seems larger for individually
administered tests, but not after control for
weeks (wkcat)
41
Individual vs. Group Tests
Order 1 to specify a 2-way interaction term
Variables in interaction
Conclusion No interaction effect (chi-sq little
different than wkcat alone (15.114 vs. 16.568).
Notice that the effect of test type (and its SE)
are very large (.2901/.4859.597)
42
Individual vs. Group Tests
Grand Mean Centered
Grand Mean Centered
Conclusion Same results but estimated SE for
individual term is smaller (reduced
multicollinearity by grand mean centering the
wkcat variable). Note that chi-sq is the same.
43
Advanced multilevel analyses
44
weeks centered at 2
Weeks is centered at 2.
gtpred c50-gtcalc c51 (('d'-c50)/'se')2 -gtsum
c51 to b1 28.937 -gtcprob b1 17 0.035115
Conclusion The effect of weeks (-.013/.005)
chi-sq (28.937) same as with original weeks. The
mean effect size (.136/.049) signif (when weeks
2).
45
weeks centered at 6 7
Weeks centered at 6
Weeks centered at 7
gtpred c50-gtcalc c51 (('d'-c50)/'se')2 -gtsum
c51 to b1 28.937 -gtcprob b1 17 0.035115
Conclusion Constant term (intercept weeks 6)
is signif (.083/.0421.98) Constant term
(intercept weeks 7) is NS (.070/.0421.68)
46
weeks polynomial 2
gtpred c50-gtcalc c51 (('d'-c50)/'se')2 -gtsum
c51 to b1 26.237 -gtcprob b1 16 0.050779
Conclusion The linear term is significant but
the quad term is not.
47
weeks polynomial 3
Conclusion All three polynomial terms are
significant and the residual variance component
is substantially reduced.
48
Log-e weeks1
Conclusion The linear term based on the log
transform explains more variance than the
original (untransformed) weeks (chi-sq 24.636
vs. 28.937).
49
WKCAT Centered at 2 3
Conclusion Intercept at wkcat 2 is
significant, but intercept at wkcat 3 is not
50
Graphs Caterpillar Plots
Caterpillar plot based on L1 residuals. Go to
the model menu and select residuals option.
This will bring up the settings window. Set SD
(comparative) to 1.96 3. Set level to 1d
4. click the Calc button 5. click on the
plot button to bring up the next window. In the
plot window select residual /- 1.96SD x
rank. This brings up the original graph. Clicking
on the graph bring up a window to modify the
graph (a bit)
51
Conclusion
52
Summary

• The mean effect size associated with the
  intervention was not significant. However, the
  results did not generalise across studies (there
  was study-to-study variation).
• Consistent with a priori predictions, the effect
  size was significantly moderated by the amount of
  time students had been in contact with teachers.
  If students and teachers knew each other 0 or 1
  week prior to the intervention, there was a
  significant expectancy effect. If they knew each
  other 2 or 3 weeks the effect was not
  significant (although the precise cutoff might
  dependend on the scaling of weeks).
• Effects of test type (individual or group) and
  test administrator awareness (blind or aware)
  were not significant and did not interact with
  length.

53
Software

• Purpose-built
• Comprehensive Meta-analysis (commercial)
• Schwarzer (free, http//userpage.fu-berlin.de/hea
  lth/meta_e.htm)
• Extensions to standard statistics packages
• SPSS, Stata and SAS macros, downloadable from
  http//mason.gmu.edu/dwilsonb/ma.html
• Stata add-ons, downloadable from
  http//www.stata.com/support/faqs/stat/meta.html
• HLM V-known routine
• MLwiN
• MPlus

54
Key references

• Cooper, H., Hedges, L. V. (Eds.) (1994). The
  handbook of research synthesis (pp. 521529). New
  York Russell Sage Foundation.
• Hox, J. (2003). Applied multilevel analysis.
  Amsterdam TT Publishers.
• Hunter, J. E., Schmidt, F. L. (1990). Methods
  of meta-analysis Correcting error and bias in
  research findings. Newbury Park Sage
  Publications.
• Lipsey, M. W., Wilson, D. B. (2001). Practical
  meta-analysis. Thousand Oaks, CA Sage
  Publications.

55
Key references

• Raudenbush, S.W. (1984). Magnitude of teacher
  expectancy effects on Pupil IQ as a function of
  the credibility of expectancy induction A
  synthesis of findings from 18 experiments.
  Journal of Educational Psychology, 76, 85-97.
• Raudenbush, S.W. and Bryk, A.S. (2002).
  Hierarchical Linear Models (2nd Ed.).Thousand
  Oaks Sage Publications.
• Download macros for free from http//mason.gmu.edu
  /dwilsonb/ma.html
• Download MLwiN for free from http//www.cmm.bristo
  l.ac.uk/MLwiN/index.shtml