Strategies for Improving

1
A CHEQ CALT Presentation
Strategies for Improving Response Rates to
Student Surveys Lorraine Bennett Chenicheri
Sid Nair Feb 2008
2
Overview

• Aim of session
• Literature Findings
• Drivers for Evaluation
• Questionnaire Structure (Unit Evaluations)
• The Process
• The Recipe
• Discussion

3
Aim

• To achieve a meaningful online survey response
  rates
• To establish an effective strategy

4
Findings I

• A criticism frequently asserted is that students
  lack the wisdom and experience to evaluate the
  effectiveness of the unit/course and teacher
• Research shows a high correlation exists between
  course-end ratings and ratings by peers,
  administrators, alumna and graduating students
  (dApollonia Abrami, 1997, Kulik McKeachie,
  1995 Centra, 1974 Drucker Remmers, 1980 Marsh
  Dunkin, 1997 Marsh Roche, 1997 ).

5
Findings II

• The assertion that student evaluation is just a
  popularity contest has been shown not to be the
  case (McKeachie, 1986 Marsh, 1987, 2001) as has
  the assertion that to gain good evaluations,
  teachers should simply make the course easy.
• In fact, Marsh (1984, 1987) reported that
  teachers who assign more and difficult work tend
  to be rated as more effective in their teaching.
  Overall and Marsh (1979) showed that teachers who
  had utilized teaching evaluations to improve
  their teaching had students who achieved higher
  test and assessment scores

6
Findings III

• Other factors generally found to be unrelated to
  student ratings include
• class size, teacher characteristics (gender,
  age, teaching experience), student
  characteristics (age, gender, personality)
• (Braskamp, 1994 Cashin, 1995 dApollonia
  Abrami, 1997 Marsh Roche, 1997, 2000)

7
Findings IV

• In short, when the total volume of research
  undertaken over the last 30 years or so is
  considered, especially the meta-analyses of this
  research, there is strong evidence that student
  evaluation provides a valid and reliable
  assessment of teaching effectiveness which in
  turn is correlated with learning gains.

8
Response Rate - Findings

• Online surveys provide a wealth of easy to use
  information from the qualitative information
  more so than paper based
• (eg. Dommeyer, Baum, Hanna, Chapman, 2004
  Hmieleski Champagne, 2000 Layne, DeCristoforo
  McGinty, 1999).
• Recent research also suggests response rates are
  higher where there is greater engagement of
  students in the process (Bennett, Nair Wayland,
  2006 Coates, 2006).

9
What is an acceptable Response Rate?

• Literature indicates that there is no magic
  formula by which a response rate can be
  identified as acceptable
• There is no evidence that online surveys with
  lower response rates produce biased evaluations
  (Coates, Tilbrook, Guthrie, Bryant, 2006
  Porter, 2004).
• Two factors that are considered important with
  regard to response rates
• - the purpose of the survey and
• - the construct of the item

10
Drivers

• Quality assurance and audit agenda.
• DEST Learning and Teaching Performance Fund
  requires evidence that student evaluation are
  conducted consistently and the results are
  publicly available on the website.
• Monash commitment to excellence in teaching.
• Monash needs to demonstrate well embedded
  evaluation policies, systems and procedures to
  ensure the capture of consistent and quality
  information on student performance and
  satisfaction levels.

11
Questionnaire Structure for Unit Evaluations

• Core Items - 10 university wide
• Faculty - Up to 10 items
• Rating Scale - Likert Scale (Strongly agree
  Strongly Disagree)
• Frequency evaluate each unit once in a year
• Accessibility - University wide items posted www

12
Questionnaire Core Items

• Quantitative
• 1. The learning objectives of this unit were
  made clear to me.
• 2. The unit enabled me to achieve the learning
  objectives.
• 3. I found the unit to be intellectually
  stimulating.
• 4. I found the resources provided for the unit
  to be helpful.
• 5. I received constructive feedback on my work.
• 6. The feedback I received was provided in time
  to help me improve.
• 7. The overall amount of work required of me
  for this unit was appropriate.
• 8. Overall, I was satisfied with the quality of
  this unit.
• Qualitative
• What were the best aspects of this
  unit?..........................
• What aspects of the unit are most in need of
  improvement?...............

13
System Design for Unit Evaluations
14
Process Timelines for Unit Evaluations
PAPER-BASED

• Week 1
• UOO for semester
• sent to faculties
• to complete
• Faculties confirm method of unit evaluation

• Week 2-5
• Evaluation Team creates questionnaire
  templates

• Week 6-9
• Evaluation Team sends templates and return
  labels to facultyadministrators
• Faculties print and distributequestionnaires

• Week 9-12
• Faculties conduct survey with students

• Week 10 -18
• Completed surveys returned to Evaluations Team
• Evaluations Teamprocesses surveys and produces
  reports

WEB-BASED

• Week 9
• Evaluations Team activates survey
• Email advice goes to students

• Week 14
• Survey deactivated

• Week 15-20
• Evaluations team processes surveys and
  producesreports

• Survey Open
• Week 11
• Week 13
• (Email Reminders)

Same As Paper Based
Same As Paper Based
15
Administration

• student evaluation of unit evaluation
  questionnaires must be distributed and collected
  by a person other than the person being evaluated

16
REPORTS - Index Page
17
Faculty Index Page
18
Sample Report
19
Strategies for Improving Response Rates

• PLAN
• Nominate evaluation coordinator for dept/faculty
• Develop communication strategy (key to success)
• Ensure leadership support (Dean/HOD, ADTs)
• Item on committees agendas (curriculum staff)
• Promote improvements in Unit Outlines/Guides,
  in class, on web
• Alert students - first session and ongoing
• Information on portals, emails (for staff and
  students)

20
Strategies for Improving Response Rates

• ACT
• Arrange for Dean/HOD presence at survey period
  (go to class occasionally)
• Arrange for Dean/HOD to send email reminders of
  encouragement
• Pre-advertise survey purpose and schedule (web)
• Place Have your Say Posters in student areas
• Monitor response rates (web-based) send reminders
• Confirm anonymity of responses

21
Strategies for Improving Response Rates

• Evaluate
• Document communication/admin procedures, note
  what worked well, problem areas
• Examine response rate time series information
  (Friday night reminders best)
• Compare response rates with other faculties
• Seek suggestions from high response rate
  faculties

22
Strategies for Improving Response Rates

• IMPROVE
• Continually refine procedures
• Clean up data
• Document and demonstrate to students that
  feedback is valued and acted upon
• Higher levels of participation achieved when
  students given feedback

23
Recent Case Study - Web
24
Discussion / Questions?