What is Metaanalysis

1
What is Meta-analysis?

• NCRM Research Methods Festival
• University of Oxford

Dept of Education, University of Oxford
2
Primary versus secondary data analysis

• Traditionally, education researchers collect and
  analyse their own data (referred to as primary
  data). Secondary data analysis is based on data
  collected by someone else (or, perhaps,
  re-analysis of your own published data). There
  are at least four logical perspectives to this
  issue
• 1. Meta-analysis -- systematic, quantitative
  review of published research in a particular
  field, the focus of this presentation.
• 2. Systematic review -- systematic, qualitative
  review of published research in a particular
  field
• 3. Secondary Data Analyses -- using large
  (typically public) databases
• 4. Re-analyses of published studies -- often in
  ways critical of the original study.

3
What is meta-analysis?

• Systematic synthesis of various studies on a
  particular research question
• Do boys or girls have higher self-concepts?
• Collect all studies relevant to a topic
• Find all published journal articles on the topic
• An effect size (the dependent variable) is
  calculated for each outcome
• Determine the size/direction of gender difference
  for each study
• Content analysis
• code characteristics of the study age, setting,
  ethnicity, self-concept domain (math, physical,
  social), etc.
• Effect sizes with similar features are grouped
  together and compared tests moderator variables
• Do gender differences vary with age, setting,
  ethnicity, self-concept, domain, etc.

3
4
A blend of qualitative and quantitative approaches

• Coding the process of extracting the information
  from the literature included in the
  meta-analysis. Involves noting the
  characteristics of the studies in relation to a
  priori variables of interest (qualitative)
• Effect size the numerical outcome to be analysed
  in a meta-analysis a summary statistic of the
  data in each study included in the meta-analysis
  (quantitative)
• Summarise effect sizes central tendency,
  variability, relations to study characteristics
  (quantitative)

5
Benefits of meta-analysis

• Compared to traditional literature reviews
• (1) there is a definite methodology employed in
  the research analysis and 
• (2) the results of the included studies are
  quantified to a standard metric thus allowing for
  statistical techniques for further analysis.
• Therefore less biased and more replicable

6
Benefits of meta-analysis

• Increased power increases the chance of
  detecting a true treatment effect
• Improved precision with more information than a
  single study, the treatment effect estimate is
  improved
• When study-to-study variation in results (which
  is typical) can evaluate differences in relation
  to study characteristics. Can delve into research
  questions not explored by the individual studies
• Easy to interpret summary statistics (useful if
  communicating findings to a non-academic audience)

7
Why is meta-analysis important? Generalisability

• The essence of good science is replicable and
  generalisable results.
• Do we get the same answer to important research
  questions when we run the study again?
• The primary aims of meta-analysis is to test the
  generalisability of results across a set of
  studies designed to answer the same research
  question.
• Are the results consistent? If not, what are the
  differences in the studies that explain the lack
  of consistency?

8
Why is meta-analysis important?

• Meta-analysis is an increasingly popular tool for
  summarising research findings literature review
  method of choice in many disciplines
• Widely-cited. If there is a good meta-analysis
  relevant to your study, you have to cite it
• Relied upon by policymakers
• Important that we understand the method, whether
  we conduct or consume meta-analytic research
• Should be one of the topics covered in all
  introductory research methodology courses

9
When is meta-analysis appropriate?

• There exists a critical mass of comparable
  studies designed to address a common research
  question.
• Data are presented in a form that allows the
  meta-analyst to compute an effect size for each
  study.
• Characteristics of each study are described in
  sufficient detail to allow meta-analysts to
  compare characteristics of different studies and
  to judge the quality of each study.

10
Schulze, R. (2007). The state and the art of
meta-analysis . Zeitschrift für
Psychologie/Journal of Psychology, 215, 87-89.
The number of meta-analyses is increasing at a
rapid rate.
11
Where are meta-analyses done?
12
Psychology Citations
Psychology Articles
13
Psychology Where it all began

• Amato, P. R., Keith, B. (1991). Parental
  divorce and the well-being of children A
  meta-analysis . Psychological Bulletin, 110,
  26-46. Times Cited 471
• Linn, M. C., Petersen, A. C. (1985). Emergence
  and characterization of sex differences in
  spatial ability A meta-analysis . Child
  Development, 56, 1479-1498. Times Cited 570
• Johnson, D. W., et al (1981). Effects of
  cooperative, competitive, and individualistic
  goal structures on achievement A meta-analysis .
  Psychological Bulletin, 89, 47-62. Times Cited
  426
• Tett, R. P., Jackson, D. N., Rothstein, M.
  (1991). Personality measures as predictors of job
  performance A meta-analytic review . Personnel
  Psychology, 44, 703-742 Times Cited 387
• Hyde, J. S., Linn, M. C. (1988). Gender
  differences in verbal ability A meta-analysis .
  Psychological Bulletin, 104, 53-69. Times Cited
  316
• Iaffaldano, M. T., Muchinsky, P. M. (1985). Job
  satisfaction and job performance A meta-analysis
  . Psychological Bulletin, 97, 251-273. Times
  Cited 263.

14
Education Widely Cited Meta-analyses

• De Wolff, M., van IJzendoorn, M. H. (1997).
  Sensitivity and attachment A meta-analysis on
  parental antecedents of infant attachment . Child
  Development, 68, 571-591. Times Cited 340
• Wellman, H. M., Cross, D., Watson, J. (2001).
  Meta-analysis of theory-of-mind development The
  truth about false belief . Child Development, 72,
  655-684. Times Cited 276
• Cohen, E. G. (1994). Restructuring the classroom
  Conditions for productive small groups . Review
  of Educational Research, 64, 1-35. Times Cited
  235
• Hansen, W. B. (1992). School-based substance
  abuse prevention A review of the state of the
  art in curriculum, 1980-1990 . Health Education
  Research, 7, 403-430. Times Cited 207
• Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980).
  Effectiveness of Computer-Based College Teaching
  A Meta-Analysis of Findings. Review of
  Educational Research, 50, 525-544. Times Cited
  198.

15
Business/Management Widely Cited Meta-analyses

• Sheppard, B. H., Hartwick, J., Warshaw, P. R.
  (1988). The theory of reasoned action A
  meta-analysis of past research with
  recommendations for modifications and future
  research . Journal of Consumer Research, 15,
  325-343. Times Cited 515
• Jackson, S. E., Schuler, R. S. (1985). A
  meta-analysis and conceptual critique of research
  on role ambiguity and role conflict in work
  settings . Organizational Behavior and Human
  Decision Processes, 36, 16-78. Times Cited 401
• Tornatzky Lg, Klein Kj. (1994). Innovation
  characteristics and innovation adoption-implementa
  tion - A meta-analysis of findings . IEEE
  Transactions On Engineering Management, 29, 28-4.
  Times Cited 269.
• Lowe KB, Kroeck KG, Sivasubramaniam N. (1996).
  Effectiveness correlates of transformational and
  transactional leadership A meta-analytic review
  of the MLQ literature. Leadership Quarterly, 7,
  385-425. Times Cited 203.
• Churchill GA, Ford NM, Hartley SW, et al. (1985).
  Title The determinants of salesperson
  performance - A meta-analysis . Journal Of
  Marketing Research, 22, 103-118. Times Cited
  189.

16
Most Widely Cited Meta-analyses are in Medicine

• Jadad AR, Moore RA, Carroll D, et al. (1996).
  Assessing the quality of reports of randomized
  clinical trials Is blinding necessary?
  Controlled Clinical Trials, 17, 1-12. Times
  Cited2008
• Boushey Cj, Beresford Saa, Omenn Gs, Et . Al.
  (1995). A quantitative assessment of plasma
  homocysteine as a risk factor for
  vascular-disease - Probable benefits of
  increasing folic-acid intakes. JAMA-journal Of
  The American Medical Assoc, 274, 1049-1057. Times
  Cited 2,128
• Alberti W, Anderson G, Bartolucci A, et al.
  (1995). Chemotherapy in non-small-cell
  lung-cancer - A metaanalysis using updated data
  on individual patients from 52 randomized
  clinical-trials. British Medical Journal, 311,
  899-909. Times Cited1,591
• Block G, Patterson B, Subar A (1992). Fruit,
  vegetables, and cancer prevention - A review of
  the epidemiologic evidence. Nutrition And
  Cancer-an International Journal, 18, 1-29. Times
  Cited 1,422

17
Classic Meta-analysis Smith Ml, Glass GV (1977)
Meta-analysis Of Psychotherapy Outcome Studies.
American Psychologist, 32, 752-760. Times Cited
840.

• Gene Glass coined the phrase meta-analysis in
  classic study of the effects of psychotherapy.
  Because most individual studies had small sample
  sizes, the effects typically were not
  statistically significant.
• Results of 375 controlled evaluations of
  psychotherapy and counselling were coded and
  integrated statistically. The findings provide
  convincing evidence of the efficacy of
  psychotherapy.
• On the average, the typical therapy client is
  better off than 75 of untreated individuals.
• Few important differences in effectiveness could
  be established among many quite different types
  of psychotherapy (e.g., behavioral and
  non-behavioral).

17
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
18
Steps in a meta-analysis
19
Defining a population of studies and finding
publications

• Need to have explicit inclusion and exclusion
  criteria
• The broader the research domain, the more
  detailed they tend to become
• Refine criteria as you interact with the
  literature
• Components of a detailed criteria
• distinguishing features
• research respondents
• key variables
• research methods
• cultural and linguistic range
• time frame
• publication types

20
Locate and collate studies

• Search electronic databases (e.g., ISI,
  Psychological Abstracts, Expanded Academic ASAP,
  Social Sciences Index, PsycINFO, and ERIC)
• Examine the reference lists of included studies
  to find other relevant studies
• If including unpublished data, email researchers
  in your discipline, take advantage of Listservs,
  and search Dissertation Abstracts International

21
Pilot coding

• Random selection of papers coded by both coders
• Meet to compare code sheets
• Where there is discrepancy, discuss to reach
  agreement
• Amend code materials/definitions in code book if
  necessary
• May need to do several rounds of piloting, each
  time using different papers

22
Develop code materials
Code Sheet
Code Book/manual

• __ Study ID
• _ _ Year of publication
• __ Publication type (1-5)
• __ Geographical region (1-7)
• _ _ _ _ Total sample size
• _ _ _ Total number of males
• _ _ _ Total number of females

ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
23
Effect size calculation

• The effect size makes meta-analysis possible
• It is the dependent variable
• It standardizes findings across studies such that
  they can be directly compared
• Any standardized index can be an effect size
  (e.g., standardized mean difference, correlation
  coefficient, odds-ratio), but must
• be comparable across studies (generally requires
  standardization)
• represent the magnitude and direction of the
  relationship of interest
• be independent of sample size

24
Effect size calculation
Means and standard deviations
Almost all test statistics can be transformed
into an standardized effect size d
Correlations
d
P-values
F-statistics
t-statistics
other test statistics
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
25
Effect sizes

• Lipsey Wilson (2001) present many formulae for
  calculating effect sizes from different
  information
• All effect sizes in a single meta-analysis must
  be in a common metric, typically based on the
  natural metric given research in the area.
  E.g.
• Standardized mean difference
• Odds-ratio
• Correlation coefficient

25
26
Constructing a database
27
Analytical methods

• There are various ways of analysing meta-analytic
  data
• Three main methods based on different statistical
  assumptions
• Fixed effects models
• Random effects models
• Multilevel models
• These will be discussed in the afternoon workshop

28
Conclusion

• Meta-analysis is a method for synthesising and
  analysing the research literature on a particular
  topic
• The essence of good science is replicable and
  generalisable results.
• Increasingly sophisticated
• Continuously evolving
• For more information about the meta-analysis
  training courses that we offer, please see
  http//education.ox.ac.uk/research/resgroup/self/t
  raining.php