Lessons Learned

1
Lessons Learned

• Findings from Ten Formative Assessments
• of Educational Initiatives at MIT
• (2000-2003)
• Lori Breslow, Ph.D.
• and
• The Staff of the
• Teaching and Learning Laboratory

2
The Context

• From 1999 to the present, MIT has undertaken
  approximately 40 experiments in educational
  innovation. TLL has assessed

Subjects 1.00, 6.001, 6.002x,
8.02 (TEAL), 8.224, HST582J, 18.03, Mission
200X
Programs The Undergraduate Exchange with
University of Cambridge, Residence-Based
Advising
Technologies PIVoT, Cross Media Annotation System
(XMAS), MetaMedia, PRS
Space The dArbeloff/TEAL classroom
(26-152)
Denotes multi-year or multi-semester assessment
3
Agenda

• Four lessons about the use of educational
  technology
• Observations on best practices in design,
  implementation, and assessment
• Where do we go from here?

4
Four Lessons about Educational Technology

• Educational technologies have contributed to
  gains in learning
• Educational technologies that have met a specific
  educational need that has been unmet or poorly
  met by traditional media have been more
  successful
• Too much new technology, too many technologies,
  technologies that are poorly integrated into the
  curriculum have been less successful
• Studying the relationship between educational
  technologies and the contexts in which they are
  embedded has yielded important understandings

5
Technology-Enabled Active Learning Increased
Learning Gains
Lesson 1

Source Dori, Y. Belcher, J. (2004) How Does
Technology-Enabled Active Learning Affect
Undergraduate Students Understanding of
Electromagnetic Concepts? under review at The
Journal of the Learning Sciences
Learning gains ltggt Correct post-test -
Correct pre-test
100 - Correct pre-test
6
A Problem-Based Module Increased Learning Gains
Lesson 1
Source Greenberg, J. Smith, N. Newman, J.
(2003) Instructional Module in Fourier Spectral
Analysis, Based on Principles of How People
Learn, Journal of Engineering Education
7
What Has Worked
Lesson 2

• Online lectures to teach students basic concepts
• Visualizations to help students see what cannot
  be seen
• Archives of visual images to help students
  strengthen different kinds of literacies
• Wireless laptops to learn programming
• Remote online laboratories
• Technologies that provide instantaneous, more
  individualized feedback

8
Online Lectures Motivate Students
Lesson 2
9
Online Lectures Were More Effective in Intro
Computer Science
Lesson 2
Source Newman, J. (2002) 6.001 Report on the
Outcomes of Online Learning, unpublished
assessment report
10
Visualizations Help Students See What Cannot Be
Seen
An animation used in Physics II, a course in
electromagnetism (8.02T). Click to see how
field lines move. For more information on the
Technology-Enabled Active Learning project, go to
http//web.mit.edu/8.02t/www.
11
MetaMedia Creates Visual Archives that Expand
Literacy
MetaMedia homepage. For more information on
MetaMedia, go to http//metaphor.mit.edu/.
12
iLab Makes Laboratory Facilities Available
Remotely
The Flagpole Project allows students to take
readings remotely. For more information on iLab,
go to http//icampus.mit.edu/projects/iLab.shtml.
13
Two Other Effective Applications of Educational
Technology
Lesson 2

• Using wireless laptops to learn programming in
  Introduction to Computers and Engineering
  Problem Solving (1.00)
• Improving feedback
• To the students through hint and check buttons
  used to answer problems embedded in online
  lectures (6.001)
• To the faculty through a personal response system
  that records student answers to conceptual
  questions asked during class (8.02T)

14
What Hasnt Worked
Lesson 3

• Technologies designed for interaction (e.g.,
  discussion boards) have been less successful at
  MIT
• When students need to learn the technology before
  they can learn the material, they have two things
  to master
• Too much technology can be detrimental
• When technology is poorly integrated into the
  curriculum, it is either not used at all or not
  used effectively by the students

15
Applications That Needed Some Work
Lesson 3

• Functionality of
• MetaMedia
• XMAS
• Simulation in Biomedical Signal Image and
  Processing
• Discussion boards in Exploring Black Holes
• Unchat in Mission 200X
• Technology in 8.02T
• PRS
• PowerPoint
• Laptops when used in lectures in 1.00

16
Edtech and the Learning Environment
Lesson 4

• Edtech has been most successful when there are
  strong connections between it and
• Learning objectives
• Pedagogies
• Assessment
• The same technology will have different effects
  in different environments
• Edtech exerts its impact by changing the nature
  of information in the system

17
There Are Strong Connections in TEAL
Lesson 4

• Objective to increase students conceptual and
  analytical understanding about the nature of
  electromagnetic fields
• Technologys contribution
• Visualizations
• Desktop experiments
• PRS

18
But the Same Technology May Have Different
Effects . . .
Lesson 4

• In different learning environments
• PIVoT as used at MIT, RPI, and Wellesley
• In different situations
• Laptops in 1.00 during active learning and during
  lecture
• For different students
• For example, less prepared MIT students were
  significantly more likely than better prepared
  MIT students to think PIVoT helped their
  conceptual understanding
• But at RPI, students with better high school
  preparation derived greater benefit than those
  with weaker preparation

19
Edtech Changes the Nature of Information in the
System
Lesson 4
Information Has Five Properties
Form
Magnitude
Velocity
Direction
Access
Source Nystrom, C. (1973) Towards a Science
of Media Ecology The Formulation of Integrated
Conceptual Paradigms for the Study of Human
Communication Systems, unpublished doctoral
dissertation
20
How Edtech Changes Information in the System and
the Impact
Lesson 4

• If form changes student literacies
  expand
• If magnitude changes students can be
  overwhelmed by information
• If accessibility and direction changes
  relationship between students and instructor
  shifts
• If velocity changes feedback can be more
  instantaneous

21
Best Practices in Design, Implementation, and . .
.

• Design
• Begin with learning objectives
• Research what is already known or has been done
  related to the innovation
• Implementation
• Estimate the amount of resources (time, funding,
  space, etc.) needed then increase it
• Assume mid-course corrections will be needed

22
... Assessment

• Assessment at MIT works best when
• It is formative
• It is collaborative, which means
• Demands on faculty time must be honored
• Differences in research in the hard and soft
  sciences must be made explicit
• Quantitative and qualitative methodologies are
  used

23
Next Steps

• The longer term impact of the earliest projects
  are being studied
• TEAL
• Mission 200X
• 6.002x
• The lessons learned will be disseminated
  through a variety of venues at the Institute and
  in the wider academic community
• The next set of research questions of interest to
  the faculty and the TLL staff will be prioritized