In Search of

1
In Search of What Works in Online and Distance
Education A Report of Two Meta-Analyses

• Robert M. Bernard, Philip C. Abrami and
• Eugene Borokhovski
• Centre for the Study of Learning and Performance
• Concordia University
• http//doe.concordia.ca/cslp

Research supported by grants from FQRSC and SSHRC
to Bernard and Abrami
2
Purposes of meta-analysis

• Estimate the population central tendency and
  variability of effect sizes between an
  intervention (treatment condition) and a control
  condition.
• Explore unexplained variability through the
  analysis of methodological and substantive coded
  study features.

3
What is an effect size?

• Standardized difference between a treatment mean
  and a control mean
• Positive if treatment gt control
• Negative if treatment lt control
• Basic Equations

4
What is a weighted effect size?

• Effect size units of analysis are samples that
  differ in their size (i.e., number of
  participants)
• Inverse variance (1/se2) sample size weighting is
  applied to each effect size so that large samples
  are counted more than small samples
• Statistical synthesis (averages g, variability
  Q-statistic, ANOVA, regression) take weighting
  into account

5
What about variability among effect sizes?
The Q-statistic is used to measure variability of
effect size (it is similar to a sum of squares
and is evaluated with the ?2 sampling
distribution)
Q-statistic is high and significant
Q-statistic is low and not significant
6
Project 1 2000 2004

• Question How does distance education compare to
  classroom instruction? (inclusive dates
  1985-2002)
• Total number of effect sizes k 232
• Measures Achievement, Attitudes and Retention
  (opposite of drop-out)
• Divided into Asynchronous and Synchronous DE

Bernard, R. M., Abrami, P. C., Lou, Y.
Borokhovski, E., Wade, A., Wozney, L., Wallet,
P.A., Fiset, M., Huang, B. (2004). How does
distance education compare to classroom
instruction? A meta-analysis of the empirical
literature. Review of Educational Research,
74(3), 379-439.
7
Summary of results Achievement
Achievement Outcomes
Type of DE k g Sig.
Combined 318 0.013 gt 0.05
Synchronous 92 0.102 lt 0.05
Asynchronous 174 0.053 lt 0.05
Significantly heterogeneous average effect
8
Summary of results Attitudes
Attitude Outcomes
Type of DE k g Sig.
Combined 154 0.081 lt 0.05
Synchronous 83 0.185 lt 0.05
Asynchronous 71 0.034 gt 0.05
Significantly heterogeneous average effect
9
Summary of results Retention
Retention Outcomes
Type of DE k g Sig.
Combined 103 0.057 lt 0.05
Synchronous 17 0.005 gt 0.05
Asynchronous 53 0.093 lt 0.05
Significantly heterogeneous effect sizes
10
Primary findings

• DE and CI are essentially equal (g 0.0 to low
  average effect) on all measures
• Effect size distributions are heterogeneous some
  DE gtgt CI, some DE ltlt CI
• Generally poor methodological quality
• Pedagogical study features account for more
  variation than media study features (Clark, 1994)
• Interactive DE an important variable

Lou, Y., Bernard, R.M., Abrami, P.C. (2006).
Media and pedagogy in undergraduate distance
education A theory-based meta-analysis of
empirical literature. Educational Technology
Research Development, 54(2), 141-176.
11
Project 2 2005 present

• Question What are the effects of three types of
  interaction in DE vs. DE studies? (Inclusive
  dates 1985 to 2006)
• Two definitions of interaction
• Interaction should refer in a restrictive manner
  to cover only those activities where the student
  is in two-way contact with another person (or
  persons) (Daniel Marquis, 1988, p. 339)
• Interactions are reciprocal events that require
  at least two objects and two actions.
  Interactions occur when these objects and actions
  mutually influence one another (Wagner, 1994, p.
  8)

12
Project 2 2005 present

• Moore (1989) distinctions are
• Three types of interaction
• student-student interaction
• student-teacher interaction
• Student-content interaction
• Anderson (2003) hypotheses state
• Deep, meaningful learning is produced from 2 out
  of 3 interactions at a high level
• High levels of more than 1 out of 3 interactions
  will produce satisfying educational experience
• Increasing satisfaction through interaction may
  not be as time or cost-effective as less
  interactive learning sequences

13
The unique problem in this meta-analysis

• In most meta-analyses, treatment and control
  designations are unambiguous (e.g., DE vs. CI)
• In DE vs. DE studies, which condition is the
  treatment and which is the control?
• Sorted 74 achievement and 44 attitude effects
  into SS, ST and SC categories
• Two judges determined which condition was the
  best type for each category that group became
  the treatment and the other the control

14
Do the three types of interaction differ? Moores
distinctions
Achievement and Attitude Outcomes
Moores distinctions seem to apply for
achievement (equal importance), but not for
attitudes (however, samples are low for SS and SC)
15
Investigating treatment strength

• Andersons hypotheses involve improving the
  strength of interaction treatments
• We defined treatment strength as ratings of
  difference between two conditions
• Coded strength as
• 0) conditions are equal
• 1) treatment is gt control
• 2) treatment is gtgt control

16
Does strengthening interaction improve
achievement and attitudes? Andersons hypotheses
Achievement and Attitude Outcomes
Andersons first hypothesis about achievement
appears to be supported
Andersons second hypothesis about satisfaction
(attitude) appears to be supported, but only to
an extent (i.e., only 5 studies in High Category)
17
Do interaction types differ in strength?
Achievement Outcomes
Attitude Outcomes
18
Do interaction combinations differ?
Achievement Outcomes
Attitude Outcomes
19
What have we learned about the effects of
interaction on achievement?

• The presence of any type of interaction enhances
  achievement outcomes
• Increasing cognitive engagement (i.e., providing
  the conditions for interaction to occur) improves
  achievement (i.e., learning)
• This is especially true for student-content
  interaction and any combination that involves
  student-content interaction
• Strengthening student-student interaction also
  appears to influence achievement

20
What have we learned about the effects of
interaction on attitudes (satisfaction)?

• The relationship between attitudes and
  interaction is more complex than for achievement
• Student-student interaction seems important
• Strengthening interactions (in general) has a
  modest impact
• The role of the teacher in DE seems to have a
  variable effect on attitudes
• There is a slight suggestion that increasing SS
  ST has an effect on attitudes

21
References

• Anderson, T. (2003). Getting the mix right An
  updated and theoretical rationale for
  interaction. IRRODL, 4(2) Online
• Bernard, R.M., Abrami, P.C., Lou, Y. Borokhovski,
  E., Wade, A. Wozney, L., Wallet, P.A., Fiset, M.,
  Huang, B. (2004). How does distance education
  compare to classroom instruction? A meta-analysis
  of the empirical literature. Review of
  Educational Research, 74(3), 379-439.
• Bernard, R.M., Abrami, P.C., Borokhovski, E., Wad
  e, A., Tamin, R. Surkes, M. (2008). Examining
  Three Forms of Interaction in Distance Education
  A Meta-Analysis of Between-DE Studies. Manuscript
  in preparation.
• Daniel, J., Marquis, C. (1979). Interaction and
  independence Getting the mixture right. Teaching
  at a Distance, 15, 25-44.
• Lou, Y., Bernard, R.M., Abrami, P.C. (2006).
  Media and pedagogy in undergraduate distance
  education A theory-based meta-analysis of
  empirical literature. Educational Technology
  Research Development, 54(2), 141-176.
• Moore, M.G. (1989). Three types of interaction.
  American Journal of Distance Education, 3(2),
  1-6.
• Wagner, E.D. (1994). In support of a functional
  definition of interaction. American Journal of
  Distance Education, 8(2), 6-26.