STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER

1
STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL
RESOURCES FOR COMPUTER INFORMATION SCIENCE
Teaching with Merlot Collection

• Emrah Orhun, Ph.D.
• Troy State University Montgomery
• Alabama, USA

2
Outline

• Introduction
• Learning and Teaching Materials in The Merlot
  Collection
• Students Satisfaction with Merlot Materials in
  CIS Classes
• Conclusions

3
Introduction

• Difficulties in locating relevant and affordable
  materials of high quality
• World Lecture Hall at the University of Texas
• 83 categories to browse and search utilities to
  locate materials
• WebCT (2002)
• 1,250 publisher course content packs in more than
  50 subject areas
• customizable materials around which to build
  courses combined with the functionality of
  WebCT's course management software.

4
Quality issue

• World Lecture Hall materials
• courses offered at accredited colleges and
  universities around the world
• WebCT packs
• editorial standards of the publisher and author
  of the text

5
MERLOT (Multimedia Educational Resource for
Learning and Online Teaching)

• Web-based open resource designed primarily for
  faculty and students in higher education
• emphasizes the quality issue

6
MERLOT repository

• Over 8,000 free learning materials and support
  materials that can be integrated within the
  context of a larger course

7
Searching and Browsing in MERLOT
8
Figure 1. Entries in the Applications
sub/subcategory of Information Technology
subcategory under Science and Technology category
9
Figure 2. Search results for TECHtionary
10
Support resources

• sample assignments
• evaluations of the learning materials
• discipline panel peer reviews
• individual user reviews

11
Peer Review Process

• Discipline communities
• Biology, business, chemistry, engineering, health
  sciences, history, information technology,
  mathematics, music, physics, psychology, teacher
  education, and world languages.

12

• Discipline teams
• (MERLOT Editorial Boards)
• selected by the MERLOT partner organizations in
  accordance with the MERLOT guidelines.
• develop evaluation standards,
• conduct peer reviews of learning materials within
  the discipline, and
• make connections with their professional
  organizations

13
Peer review process

• Evaluated by at least two team members who have
  relevant expertise.
• Review process stages
• triage review to identify worthy candidates
• review of candidates
• composite review report sent back to the authors
  for feedback and permission to post the review on
  MERLOT.
• approved review is posted to the MERLOT Web site
  by the discipline leader

14

• The evaluation criteria
• quality of content,
• potential effectiveness as a teaching tool,
• and ease of use.

15

• Each dimension evaluated separately (on a scale
  of 1-5 stars, with 5 being the highest), and an
  overall rating is computed
• A review must average three stars in order to be
  posted to the site.

16
Figure 3(a). Peer Review of TECHtionary (Part a)
17
Figure 3(b). Peer Review of TECHtionary (Part b)
18
Figure 3(c). Peer Review of TECHtionary (Part c)
19
Figure 4. Member Comments for TECHtionary
20
Current state of MERLOT
21
Table 1 Growth of MERLOT materials, members, and
peer reviews (May 28, 2002)
  Total Added during last 365 days Added during last 30 days
Materials 6863 1991 229
Members 8613 4142 600
Peer Reviews 479 361 20
22
Table 1 Growth of MERLOT materials, members, and
peer reviews (February 27, 2003)
Total Added during last 365 days Added during last 30 days
Materials 8331 2338 284
Members 12637 5527 712
Peer Reviews 778 86 2
Member Comments 1790 305 18
Assignments 322 59 1
23
Table 1 Growth of MERLOT materials, members, and
peer reviewsfor IT Category
Total Added during last 365 days Added during last 30 days
Materials 559 274 21
Members 524 259 30
Peer Reviews 36 3 0
Member Comments 218 99 6
Assignments 11 9 0
24
Ratio of Peer Reviews, Member Comments, and
Assignments to Materials
May 25, 2002 September 26, 2002 February 27, 2003
Materials 6863 7395 8331
Peer Reviews 7.0 8.5 9.3
Member Comments 21.2 20.8 21.4
Assignments 3.4 3.8 3.9
25
Problem

• Percentage of reviewed materials low

Cause

• Editorial peer review process too slow

Potential Solution

• Peer review by community of learners

26
STUDENTS SATISFACTION WITH MERLOT MATERIALS IN
CIS CLASSES
27
Goal of Study

• To find out to what extent MERLOT provides
  relevant and quality materials for the CIS
  courses.

28
Questions

• Is the quality and quantity of learning materials
  in the MERLOT collection sufficient to satisfy
  CIS students knowledge/learning needs in
  specific courses?
• Is the MERLOT collection usable by the
  students?
• Does the MERLOT collection induce the students to
  become members of learning communities?
• Can peer review by students substitute for peer
  review by editorial boards?

29
Questions re-formulated

• How do students rate their satisfaction with
  MERLOT collection?
• How do students rate the learning materials as
  reviewers? Do students ratings of the learning
  materials differ from the ratings given by
  Discipline Panel members?

30
Teaching with the MERLOT Collection in CIS
Courses

• Summer-Fall 2002 semesters in the following CIS
  classes at Troy State University Montgomery
  
• CIS 5547 CAR Applied Systems Analysis
• CIS 5547 CBR Applied Systems Analysis
• CIS 3332 Software Engineering I
• CIS 3357 Logical Structures of Computer
  Design
• Overall, 41 students participated in the
  evaluation. A total of 252 evaluations were made.

31
 
Student Assignment

• Evaluation of web-based learning materials  
•  
• 1.  Visit MERLOT (www.merlot.org) site and
  register as a member.
• 2.       For each chapter covered in the course
• a.      Find a relevant learning material in the
  IT subcategory.
• b.      Review the material using the guidelines
  provided by MERLOT. In the table provided below,
  state what learning outcomes are supported by the
  material and post your review as a Member Comment
  on MERLOT.
• c.      Rate your satisfaction with MERLOT
  collection on a scale 1-5, with 5 implying that
  you were fully satisfied with the quality and
  quantity of learning materials you found in
  connection with your task, and 1 implying that
  you were not satisfied at all.
• d.      Find a relevant online learning material
  not contained in MERLOT, post it to MERLOT and
  add a Member Comment.

32
Example of Relevant MERLOT Material
Chapter Topic Title URL Learning outcome Satisfaction with MERLOT collection for this task 5 very good 1very poor
1 Players in the System Game Enterprise Resource Planning Supersite http//www.erpsupersite.com Provides analysis on ERP trends 3
2 Information System Building Blocks Interface Design for Educational Multimedia http//www.cdl.edu.rachel/PDF/intdesmm.pdf Describes the principle of interface design 4
3 Information System Development Risk analysis and management for Projects (RAMP) http//www.ramprisk.com/frontindex.htm Comprehensive outline of RAMP  5
33
Example of New MERLOT material posted
Chapter Topic Title URL Learning outcome
1 Players in the System Game New responsibilities Fall Heavy on IT Workers www.informationweek.com Talks about the new job responsibilities of current and upcoming IT employees.
2 Information System Building Blocks TP Middleware keeping up with Mainframe Revival www.adtmag.com Explains the functionality of the product TP middleware on mainframe systems.
3 Information System Development Tips for a successful Software Distribution Project. www.computerworld.com Shares other companies success stories of in relation to Software Distribution Project.
34
How do students rate their satisfaction with
MERLOT collection?
35
Students mean satisfaction ratings for MERLOT
collection
36
CIS3357 Students satisfaction with MERLOT
collection
37
CIS3332 Students satisfaction with MERLOT
collection
38
CIS5547 Students satisfaction with MERLOT
collection
39
CIS5547 Students satisfaction with MERLOT
collection (contd.)
40
How different are students evaluations from
Editorial Peer Reviews?
41
Distribution of materials in IT
Type
Tutorial File System Design for an NFS File Server Appliance   Cascading Style Sheets, Level1 Electronic Commerce Tutorial   FTP Tutorial Learning Microsoft Word 2000   Web-Database Integration   Mulder's Cascading Style Sheets Tutorial   Sharky's Netscape Frames Tutorial   Tje ABCs of Building a Web Site Data Communications   ADSL   Computer Knowledge Virus Tutorial
Reference Material Study Guides and Strategies Tips for Using Blackboard
Animation TECHtionary Warriors of the net
42
Sample of Materials from IT Category
Material Type
Tutorial     ADSL
Reference Material Web-Database Integration
Animation TECHtionary  
Distribution of materials according to type and rating (excluding materials with no user comments with 3 categories)
43
Sample of Students

• CIS3332 SOFTWARE ENGINEERING (16)
• CIS5547 SYSTEMS ANALYSIS (27)

44
Students Ratings
Case Editorial Board Rating Member Rating Students Rating (mean)
TECHtionary 3.0 3.75
Web-Database Integration 3.67 3.82
45
Students ratings for TECHtionary
Mean Std. Dev. Std. Error Variance Coef.Var. Count
3.754 .421 .086 .177 11.218 24
Minimum Maximum Range Sum Sum Squared Missing
2.5 4.4 1.9 90.1 342.33 0
Editorial Board Rating 3.00
46
Students ratings for Web-Database Integration
Mean Std. Dev. Std. Error Variance Coef. Var. Count
3.829 .545 .119 .297 14.238 21
Minimum Maximum Range Sum Sum Squared Missing
2.8 4.9 2.1 80.4 313.76 0
Editorial Board Rating 3.67
47
Is the mean student rating different from the
rating given by the Editorial Board?

• TECHtionary
• Different (one-group t-value 8.773, p0.0001)
• Web-Database Integration
• Insufficient evidence (one-group t-value 1.333,
  p0.1975)

48
Conclusions

• Students are satisfied with the quality and
  quantity of learning materials they found in the
  IT category.
• Current data suggest that students rate materials
  differently than editorial boards.
• The present limited study does not provide
  evidence for the notion that peer review by
  students can substitute for peer review by
  editorial boards.