Online Course Ratings In Higher Education

1
Online Course RatingsIn Higher Education

• Trav Johnson, Brigham Young University
• Michael Theall, Youngstown State University

2
Use of Online Ratings

• How many institutions use online course ratings?
• Approximately 120 colleges and universities have
  been identified as using online student ratings
  in some capacity (http//OnSET.byu.edu).
• The use of online ratings is increasing rapidly
  the actual number is probably many more than 120.

3
Use of Online Ratings

• To what extent do these institutions use online
  course ratings?
• 16 have campus-wide online course rating systems.
• 43 use online ratings for at least an entire
  department but less than the whole campus.
• 34 use online ratings for online and distance
  courses only.
• 25 use online ratings for less than an entire
  department.
• At least 2 institutions provide online reports of
  student rating data that is collected using
  paper-pencil forms.

4
OnSET Websitehttp//OnSET.byu.edu

• Information on online ratings is being collected
  and periodically updated on the Website for
  Online Student Evaluations of Teaching (OnSET).
• This Website is provided as a service to those
  who are interested in learning about and sharing
  information on online ratings.

5
What benefits of online ratings has your campus
experienced (or would you anticipate
experiencing)?
6
Some Possible Benefits ofOnline Ratings Over
Paper Ratings

• Quicker feedback to instructors Online data
  collection can potentially provide reports at any
  time they are needed after the rating period
  closes.
• Anonymity of student comments Faculty cannot
  identify student handwriting.
• Longer and more thoughtful student comments
  Student have more time to complete rating forms
  and make comments.

7
Some Possible Benefits ofOnline Ratings Over
Paper Ratings (cont.)

• Class-time savings Time in class is not needed
  for students to complete rating forms.
• Flexibility in rating form and reports Easier to
  provide customized rating forms and reports.
• Cost savings over time Although there is usually
  a large initial cost, there may be cost savings
  in the long run.

8
What challenges related to online ratings has
your campus experienced (or would you anticipate
experiencing)?
9
Challenges in Implementing and Using Online
Ratings

• Faculty and student support
• Initial costs for system development
• Student confidentiality
• Response rates
• Appropriate access to, and use of, online reports

10
Faculty and Student Support

• Students must choose to complete the online
  ratings on their own time outside of class.
• Faculty need to promote (or at least support) the
  online rating system.
• If the ratings arent important to the faculty,
  they wont be important to students, resulting in
  students not completing the ratings or not taking
  them as seriously as they would otherwise.
• If faculty members dont support the system, they
  wont use the results to improve teaching (and
  they may even work to sabotage the system!).

11
Initial Cost

• Computing resources (e.g., hardware, software,
  networks, security)
• Staff expertise
• Conceptualization and design
• Computer programming
• User support
• Education (e.g., validity/reliability of student
  ratings generally and online ratings
  specifically, how to use the system, interpreting
  reports)
• Promotion and advertising

12
Student Confidentiality

• Student identification is required when students
  submit an online rating form.
• Students must be confident that their identity is
  kept strictly confidential, especially in regard
  to access by faculty members.
• Separate student identifiers and student
  responses at time of rating form submission.

13
Response Rates Low Response Rates and
Non-Response Bias

• Non-response bias is when survey respondents
  provide different responses than would have been
  provided by those who did not respond.
• In theory, the larger the percentage of
  non-respondents, the more likely non-response
  bias will occur.
• Does non-response bias occur when student rating
  response rates are low?

14
Study 1 Paper and Online Ratings Administered in
74 Course Sections

• The online results were on average 0.1 point
  higher than the paper ratings, even though the
  online response rates were typically lower.
• For paper ratings, the correlation between
  response rates and overall ratings was .41
• For online ratings, the correlation between
  response rates and overall ratings were .09 and
  .10

15
Study 2 Paper and Online Ratings Administered in
91 Course Sections

• High correlation between paper and online results
  (0.87, 0.89), even though the same students
  didnt necessarily complete both forms.
• For online ratings, the correlation between
  response rates and overall ratings was not
  statistically significant.
• For online ratings, point estimates suggest that
  it would take a reduction in the response rate of
  about 40 percentage points to lower the overall
  course or instructor rating by 0.1

16
Does this mean that response rates shouldnt be a
concern?

• No.
• These studies provide preliminary data and
  general statistics. If response rates are low,
  response bias can still be a problem in a
  particular course.
• Sufficient response rates are important in
  determining overall reliability and validity.
• Nevertheless the data suggest that online ratings
  may be less susceptible to non-response bias than
  paper ratings.

17
Increasing Online Response Rates Some Strategies

• Instructors encourage students to complete the
  ratings and let students know that they are
  interested in and use rating results.
• Student who complete ratings are entered into a
  prize drawing (e.g., iPods, laptops, bookstore
  certificates).
• Campus-wide advertising/publicity campaigns

18
Current BYU Publicity Design
19
Increasing Online Response Rates Some More
Effective Strategies

• Those students who complete their ratings can
  view their grades early online.
• Those students who complete their ratings can
  view online the rating results for all courses.

20
Appropriate Access to, and Use of, Online Ratings
Reports

• Who should have access?
• What level or type of access should each person
  have (e.g., which reports, what portion of the
  reports)?
• How can online reports be appropriately
  customized to each individual who has access to
  them?
• How are confidentiality and security of faculty
  data ensured?

21
Increasing Appropriate and Effective Use of
Online Reports

• Graphics, pop-up windows, help pages
• Interpretation and usage guidelines
• Specialized reports
• Links to information on student ratings (e.g.,
  research, theory, appropriate and effective use
  of rating results)
• Links from online rating reports to faculty
  development resources http//studentratings.byu.e
  du and http//www.byu.edu/fc/pages/tchlrnpages/Ite
  mResources.pdf

22
Resources for Development and Use of Online
Rating Systems

• http//OnSET.byu.edu (general Website with
  information about online ratings)
• Information about various online ratings systems
• http//studentratings.byu.edu (BYU online
  system), also see http//fc.byu.edu/New/tpages/fee
  dback/onlinestudentratings.html
• http//www.cmu.edu/fce/faculty/ (Carnegie Mellon
  U. online system)
• http//www.webs.uidaho.edu/studentevals/ (U. of
  Idaho online system)
• http//www.oir.uiuc.edu/dme/eon/index.htm (U. of
  Illinois online system)
• http//www.washington.edu/oea/services/course_eval
  /ias_online/index.html (U. of Washington online
  system)
• http//www.tltgroup.org/Beta/betahome.htm (TLT
  BeTA project, Better Teaching through Assessment)
  and http//www.tltgroup.org/Beta/studentengage.htm
  (information about online rating response rates)
• http//www.idea.ksu.edu/StudentRatings/online_inde
  x.html (IDEA online rating system)

23
Resources for Development and Use of Online
Rating Systems (cont.)

• Sorenson, D.L., Johnson, T.D. (Eds.) (2003).
  On-line student ratings of instruction. New
  directions for teaching and learning 96. San
  Francisco Jossey-Bass.
• Theall, M. Franklin, J. (2001) Using technology
  to facilitate evaluation. In C. Knapper P.
  Cranton (Eds.) Fresh approaches to the
  evaluation of teaching. New Directions for
  teaching and learning 88. Winter. San Francisco
  Jossey Bass.