Choosing Critical Indicators in Online Learning Evaluation

1
Choosing Critical Indicators in Online Learning
Evaluation

• Mark Hawkes, Dakota State University
• Merrill Chandler, University of Illinois
• American Evaluation Association
• Annual Conference, November 8, 2001

2
Presentation Objective

• Discuss online learning evaluation approaches in
  graduate programs at two universities
• Identify criteria/indicators suitable for the
  evaluation of online learning environments

3
Distance Learning Literature
Evaluation . . . . . . . No Yes
(21)
(79)
Focus . . . . . . . . . Training Education
(87)
(13)
Impact on Learning . . . . . . . No Yes
(58)
(42)
4
OnlineLearning Architecture
Instructor
Student
Student
Student
Student
Student
Internet
Login Interface
Content
Resources
Management and Organizational Information
Assessment
Communication Modes
Support Services
System Resources
5
Interaction Example
Performance
Catalog info
Query
Preferences
(Metcalf, Snitzer, Austin, 2001)
6
Familiar Online Learning Evaluation Targets . . .

• interface design
• instructional design
• student satisfaction
• technology access
• faculty satisfaction
• economic viability
• departmental capacity
• interdepartmental collaboration

7
DSUs Educational Technology Program

• Students
• 36 Credit hour MS program
• 80 Education 20 Business/industry
• 90 Online 10 On campus
• Female 68 Male 32
• Project-based curriculum

8
DSUs ET Environment

• Pervasive technological culture
• Consistency between program goals and the
  state/region-wide initiatives
• Campus-wide faculty support
• Institutional experience in Web-based instruction
  delivery
• Multi-delivery methods
• Client teachers, teacher developers, trainers,
  technology coordinators, etc.
• Predominantly web-based delivery

9
An Evaluation Model . . .

Illuminative Operation of Components And
Subcomponents
Integrative Holistic perspective on The learning
experience
Components
Course Program Design
Infra- structure /System
Work Flow
Interaction
Impact
Process Impact
Observing and Detecting Focused on
Performance Functional Problems
Outcomes
10
Infrastructure/System

• Input/output devices
• Network speed and connectivity
• Network design/Topology
• Technical support systems and maintenance

11
Course and Program Design

• Nature of the Design
• Situation Based
• Role of State and National Standards
• Sequencing/Instructional Strategies
• Assessment
• Motivation Learning vs. Performance
• Visualization Tools and Media
• User Interface
• Course Management

12
Work Flow

• Use of discussion tools
• Software usage
• Message redundancy (audio, video, web pages,
  emails).
• Progression
• Do learners progress through their work tasks in
  a linear fashion? (novice-like)
• Nonlinear opportunistic fashion (expert-like)

13
(No Transcript)
14
Interaction

• Social and instructional
• Must account for all of the following
  relationships

Instructor
Learner
Learners
Learners
Technology
Technology
Content
Content
15
Online Course Interaction

• Announcements
• Email
• Discussion Board
• Synchronous text chat
• Desktop Video
• File Loading
• Online assessment
• Audio/video clips
• Room-based Video

16
Impact

• Course performance
• Collaborative learning
• Retention/attrition (course and program)
• Professional relevance and utility
• Learner productivity

17
Evaluation Attributes

• Multi-sourced data (students, server log files,
  etc)
• Internal and external
• Performance based
• Comparison and criterion based

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
The breadth of this course was
1 2 3 4 5 6 7 Not nearly enough The
right amount Way too much Compared to a
traditional course 1 2 3 4 5 6 7 A much narrower
range About the same range A much wider
range of of material was covered of material
was covered material was covered Online
4.61 Compared to traditional 4.65 n32
22
The depth of this course was
1 2 3 4 5 6 7 Not nearly enough The
right amount Way too much Compared to a
traditional course 1 2 3 4 5 6 7 Material was
covered in Material was covered in Material
was covered in much less depth about the
same depth much more depth Online
4.48 Compared to traditional 4.42 n32
23
The extent of critical thinking required
1 2 3 4 5 6 7 Not nearly enough The
right amount Way too much Compared to a
traditional course 1 2 3 4 5 6 7 Much
less About the same
Much more Online 4.61 Compared to traditional
4.94 n32
24
The amount of effort put into the course
1 2 3 4 5 6 7 Much less About
the same Much more Compared to a
traditional course 1 2 3 4 5 6 7 Much
less About the same Much
more Online 5.65 Compared to traditional
5.26 n32
25
U of Is Curriculum Technology and Education
Reform (CTER)

• Master of Education (Ed.M.)
• For practicing K-12 teachers and administrators
• A two-year program
• Eight online courses
• Project based

26
CTER . . .

• Program is in its fourth year
• CTER cohorts 1 2 have graduated
• CTER 3 cohort has 26 students
• CTER 4 has cohort 25 students
• Female 73 Male 27
• Many students have technology responsibilities
  for their schools or districts

27
CTERs synchronous and asynchronous technologies

• WebBoard conferencing
• Streaming media using Real Player
• Audio narrated PowerPoint presentations
• Tapped In
• CTER Base
• iVisit
• RogerWilco
• Interactive Multimedia Paper

28
CTER evaluation

• Mostly formative
• Mixed methods
• Course evaluation
• Program evaluation
• Mini-case studies

29
Course Evaluation

• Instructor and Course Evaluation System (ICES)
• Piloting Evaluation Online (EON)
• CTER course survey using SurveyIt
• Instructor
• Technology use
• Support
• Exemplary student projects

30
Program Evaluation

• Program surveys
• Application skills
• Web browser skills
• Learner profile
• Student interviews
• Collection of student artifacts
• Mini-case studies

31
CTER studies identify five dimensions of
effective learning

• Relevant and challenging assignments
• Providing adequate and timely feedback through
  teacher-student interaction
• Flexibility in teaching and learning
• Constructing coordinated learning environments
• Constructing rich environments for student to
  student interaction

32
Indicators of CTER effectiveness

• Low dropout rate
• Student satisfaction
• Student learning transferred into practice

33
Typical Problems with Online Courses

• Facilitating and encouraging collaboration
• Time management
• Student proficiency with course tools
• Ambiguous directions
• Timeliness of feedback

34
Factors Beyond ID Control

• Student sophistication with technology tools
• System capacity
• Learner availability/accessibility
• Enthusiastic, responsive instructor
• Good learner support
• Motivated learners

35
How to Design and Effective Online Course?

• Follow basic ID principals
• Build a climate of disclosure and full
  participation
• Institute informal student evaluation and
  check-in mechanisms
• Active and intensive instructor participation
• Build in as much interactivity as possible
• Create visually interesting screens/pages
• Ensure instructions are very clear
• Multi-mode interaction is critical

36
Slides at

• www.homepages.dsu.edu/hawkesm/