Engaging the Student Voice to Enhance Student Services

1
Engaging the Student Voice to Enhance Student
Services

• Dr Hamish Coates
• coatesh_at_acer.edu.au
• November 2006

2
Focus of the presentation

• A strategic review of educational data
• Nature and use of existing data
• Student engagement A lens for analysis
• A tool for analysis CEQuery
• What can be done?

3
Quality data for management

• The quality of education data underpins the
  cogency of many strategic and operational
  decisions
• Senior managers require timely access to
  informative and robust data
• But often the wrong data is collected
• But much data management within Australian higher
  education institutions is ad hoc
• ? We need to think strategically about what data
  is required for institutional management
• We need efficient systems for analysing and
  interpreting large amounts of complex data

4
Identifying existing data

• Does the institution have central data management
  protocols?
• What data exists centrally, and within faculties,
  departments, units, subjects?
• What is the nature and limitations of existing
  data?
• What is the strategic or operational relevance of
  the data?
• How have the data and collections been developed?

5
Reviewing data needs

• What is required to bring disparate data together
  initially, and in routine ways?
• How is this data best analysed and reported, and
  who should do this?
• What are the relevant points of reference norms
  or benchmarks/criteria?
• What gaps exist in the data collection?
• What redundancies exist, and what are the legacy
  issues?

6
Focus of the presentation

• A strategic review of educational data
• Nature and use of existing data
• Student engagement A lens for analysis
• A tool for analysis CEQuery
• What can be done?

7
A summary of existing data
8
Internal student support data
9
External student support data

• CEQ Learning Resources Scale (8/38 institutions)
• The library resources were appropriate for my
  needs
• The study materials were clear and concise
• It was made clear what resources were available
  to help me learn
• Course materials were relevant and up to date
• Where it was used, the information technology in
  teaching and learning was effective
• CEQ Student Support Scale (18/38 institutions)
• I was able to access information technology
  resources when I needed them
• Relevant learning resources were accessible when
  I needed them
• Health, welfare and counselling services met my
  requirements
• The library services were readily accessible
• I was satisfied with the course and careers
  advice provided

10
Data use by institutions
11
Focus of the presentation

• A strategic review of educational data
• Nature and use of existing data
• Student engagement A lens for analysis
• A tool for analysis CEQuery
• What can be done?

12
Exploring student engagement

• Why take the engagement approach?
• How can we factor insights on student
  engagement into quality management?
• Why isnt student engagement information implicit
  in quality educational management?
• Who defines how students should interact with
  universities?
• Who benefits from evidence-based understandings
  of engagement?
• What value could engagement insights add to
  higher education?

13
Educational indicators
14
(No Transcript)
15
The idea of student engagement

• Individuals learn and develop through behavioral,
  cognitive and affective involvement with key
  educational and social practices
• People learn when staff and institutions provide
  supports likely to encourage involvement
• A specific idea which encompasses a broad range
  of relevant academic and non-academic phenomena
• Measures of engagement provide an index of
  whether students are engaging with university in
  ways likely to stimulate development and generate
  high quality learning
• Student engagement is not satisfaction, not
  retention, not just/primarily the evaluation of
  pedagogy, not an attitudinal/opinion survey, and
  not student experience

16
Engagement as a development

• Developed from interpretations of individual
  learning and development as constructive
  participation in university communities
• Astins involvement principle
• Paces quality of effort
• Tintos work on integration and departure
• Based on the identification of activities and
  conditions linked empirically with effective
  learning
• Chickering Gamsons seven principles
• identification of process factors by Pace,
  Pascarella Terenzini, Astin, and Ewell

17
Engagement as a reaction

• Engagement offers a valid and consistent measure
  of the quality of education
• Limitations of alternative means of evaluating
  the quality of university education
• institutional resources and reputations
• measures of teaching quality and teacher
  qualifications
• student input, progression and outcomes

18
The significance of engagement

• Reflects students intrinsic involvement with
  education
• Offers a generalisable indirect measure of
  educational outcomes
• A direct measure of involvement in key processes
• Focuses quality considerations on student
  learning and development
• Reflects the wide range of educationally
  meaningful interactions that students have with
  their universities
• A coincident measure of student activity that can
  be used to evaluate and manage the quality of
  higher education

19
Typical qualities of student engagement

• Constructive teaching
• Supportive learning environment
• Teacher approachability
• Student and staff interaction
• Academic challenge
• Active learning
• Complementary activities
• Collaborative work
• Beyond class collaboration
• Online forms of engagement

20
Styles of engagement
21
Possible measurement approaches

• One-to-one interaction
• Direct naturalistic observation
• Administrative data
• Time or activity diaries
• Assessment
• Interviews and focus groups
• Questionnaires and surveys

22
Student experience instruments

• College Student Experiences Questionnaire (CSEQ)
• College Student Expectations Questionnaire (CSXQ)
• College Student Survey (CSS)
• Freshman Survey (FS)
• Your First College Year (YFCY)
• National Student Survey (NSS)
• First Year Experience Questionnaire (FYEQ)
• Course Experience Questionnaire (CEQ)
• Postgraduate Research Experience Questionnaire
  (PREQ)

23
Student engagement instruments

• College Student Report (CSR)
• Student Engagement Questionnaire (SEQ)
• (CEQuery)

24
Levels of engagement
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
Using engagement data

• Guidance for prospective students on
  expectations, experiences and supports at an
  institution
• Data on resource use, management and educational
  value
• Information for internal and external quality
  enhancement activities
• Insight for graduate employers on individuals
  involvement with their work
• Feedback for teaching academics to help them
  manage and improve pedagogy and learning
• Advice for current students on course quality and
  learning practices, retention and involvement

30
Focus of the presentation

• A strategic review of educational data
• Nature and use of existing data
• Student engagement A lens for analysis
• A tool for analysis CEQuery
• What can be done?

31
CEQuery context

• Graduate Careers Australia conducts an annual
  survey of graduates using the Graduate
  Destinations Survey (GDS) and the Course
  Experience Survey (CEQ) these are sent out
  together
• These surveys capture a range of demographic
  variables linked to responses on graduate
  outcomes (GDS) and satisfaction (CEQ)
• The CEQ also has sections for written comments
  under the headings Best aspects and Needs
  improvement these are analysed by CEQuery
• This enables the students qualitative comments
  to be linked with demographic and field of
  education variables, satisfaction scales and
  graduate outcomes

32
The opportunity and need

• Need not just to gain but retain students
• Rapid developments in I.T.
• 750,000 CEQ comments - unanalysed
• Listen to the voice of students and not just
  rely on quantitative measures

33
What is CEQuery?

• IT-enabled analysis of large amounts of
  qualitative data on HE learning teaching
• Wide range of custom analyses against any of the
  CEQ student demographic categories or items
• Results in tabular, graphical or digital modes
• Allows user to test the veracity of the coding by
  showing the comments that made up the count
• Five domains and 26 subdomains

34
CEQuery coding domains

• Assessment
• Course design
• Outcomes
• Staff
• Support

35
CEQuery subdomains

• Course design
• Flexibility
• Learning methods
• Practice-theory links
• Relevance
• Structure

• Assessment
• Expectations
• Feedback
• Marking
• Relevance
• Standards

36
CEQuery subdomains

• Outcomes
• Further learning
• Intellectual
• Interpersonal
• Personal
• Knowledge/skills
• Work application

• Staff
• Accessibility
• Practical experience
• Quality and attitude
• Teaching skills

37
CEQuery subdomains

• Support
• Infrastructure
• Learning resources
• Library
• Social affinity
• Student administration

• Student Services
• Course administration

38
Hit Rates Domain Level
39
(No Transcript)
40
Staff, Teaching Skills, Needs Improvement

• Some lecturers provided poor lectures and in some
  subjects the questions in the final exam were
  nothing like the ones we had been taught to
  answer.
• Affordable technology needs to be better
  -utilized as teaching aids by staff who know that
  teaching is more than standing up the front
  droning on
• Lecturers need to be brought in line now
  traditional teaching methods are becoming
  redundant to tutorials. They need to learn how to
  actively involve us, get us working creatively,
  experimenting less chalk and talk.
• The teaching staff should give more constructive
  feedback and make learning more interactive
  during classes.
• The methods employed by lecturers are boring they
  need to participate more with audience to
  encourage participation in discussions and
  enhance interest.

41
Validity of the results

• CEQuery comments hit rate 80
• Of the remaining 20 not hit in CEQuery, half are
  not meaningful (cannot be coded)
• Coding of comments to sub-domains by CEQuery
  checked by multiple raters
• Accuracy approx 90
• CEQuery dictionary of words and phrases has been
  refined and updated, and custom dictionaries have
  been produced

42
A review of CEQuery

• Geoff Scott (2005). Accessing the Student Voice.
  DEST Canberra.
• Analysed comments made by 95,000 graduates
  between 2001-4 at 14 universities
• Key insights into CEQuery use, correlates of
  response, key aspects of engagement, best
  aspects and improvement areas
• Findings validated through a series of national
  workshops

43
Perceptions of quality by subdomian
44
Importance perceptions by subdomian
45
Clusters of the best learning methods
46
Clustered best learning activities in Business

• Face to Face
• Lecture, group project, tutorial, class-work
  exercise, discussion and sharing ideas, seminar
  and individual presentation, workshop, debate,
  11 mentor, conference, symposium, forum,
  exhibition, peer assistance, group dynamics
  exercises, critique of student production, buzz
  group
• Independent Study
• Completing assignments, thesis, self
  teaching materials, self-directed study, project
  report writing, learning contract
• Simulations Labs
• Mock trial, role play, hypothetical, games,
  experiments, labs, simulation

• Practice oriented/real world
• Clinical placement, practicum, practical
  experience, PLT, Coop Ed, work experience, hands
  on practice, field study/site visit, practical
  work,camps, real world problems, guest
  practitioners, professional mentor, design
  studio, artistic production, study abroad, case
  study
• ICT-enabled learning
• Online search, web-based learning, email,
  phone, teleconference, CD, DVD, tele-tutorial,
  radio, TV, digital image access, computer
  assisted assessment, simulations, computer
  generated demonstrations of processes,
  interactive programs

47
Most common best aspects by field of education

• Science Built Environment
• Team/group project, assignments, field
  study/site visit, hands on practice, lecture,
  class exercises, laboratory work, practical work,
  practical experience
• Health
• Clinical placement, practical experience,
  lecture, hands on practice, assignments,
  tutorial, class exercises, group project work,
  labs, practicum, work experience
• Education
• Practicum, practical experiences, assignments,
  hands on practice, lecture, tutorial, class
  discussion, class exercises, team/group project
  work

• Management Commerce
• Team/group project, assignments, lecture, class
  exercises,, seminar individual presentation,
  tutorial, discussion, case study, real world
  problems to solve, work experience
• Society, Culture Creative Arts
• Assignments, class exercises, lecture, tutorial,
  group project, class discussion, hands on
  practice, practical experience, seminar
  individual presentation, practical work

48
CEQ scales and CEQuery domains

• All five domains significantly associated with
  all CEQ scales
• Best aspects comments predict higher CEQ scale
  scores and needs improvement comments lower CEQ
  scale scores
• Strongest correlation with the Good Teaching
  Scale, Overall Satisfaction Item (OSI) and Clear
  Goals and Standards Scale
• For example given OSI score, odds of positive
  rather than a negative comment given OSI score
  Assessment 1.22, Course Design 1.34, Outcomes
  1.40, Staff 1.21, Support 1.21 (eg positive
  support comment 21 more likely for each step
  up 5-point CEQ scale)

49
(No Transcript)
50
Conditions that enhance engagement

• A sound, responsive, flexible, relevant, clear
  and mutually reinforcing course designa design
  that uses an appropriate variety of interactive,
  practice-oriented and problem-based learning
  methods
• Capable, committed, accessible and responsive
  staff being in place to deliver and improve the
  design during implementation
• Efficient and responsive administrative, IT,
  library and student support systems actively
  working together to support its operation
• Relevant, consistent and integrated assessment of
  a university standard that the courses design,
  learning methods and resources specifically
  enable students to complete

51
Key findings

• It is the total experience that retains students
• Graduates see certain experiences are more
  important than others, and identify high
  quality areas as well as those in need of
  improvement
• Areas with highest hits identify key QA
  checkpoints for course design, accreditation and
  review
• Variation between field of education and
  institution suggests opportunities for
  benchmarking
• Should address the areas with low odds of a best
  aspect comment (assessment, course structure,
  support)
• Follow up specific issues, such as the low level
  mention of ICT as a best aspect

52
Focus of the presentation

• A strategic review of educational data
• Nature and use of existing data
• Student engagement A lens for analysis
• A tool for analysis CEQuery
• What can be done?

53
Using findings to enhance practice

• Student engagement data can be used to enhance
  the quality of education and the university
  experience
• Strategic use of ICT/LMS/portals in ways that
  supplment and enhance the student experience
• We need to develop measures of engagement, and
  make better use of the CEQuery tool
• We need to develop ways of factoring
  evidence-based perspectives on student engagement
  into practice

54
Use CEQuery

• Available to all universities
• Analyse feedback data from many survey
  instruments
• Can be tailored for local contexts
• Need to enter open-text response
• Need to position within strategic reporting and
  planning cycles
• Identify good practice, internal benchmarking,
  priorities for improvement

55
Review student engagement

• Ensure feedback instruments are harnessing data
  on what really matters for high quality learning
  and support were not measuring key aspects of
  higher education
• Ensure analysis and reporting approaches are
  appropriate
• Weave student engagement ideas into operational
  and strategic plans, and practices

56
Help students learn to engage

• Produce resources which help students understand
  key aspects of being a university student
• Develop targeted programs and spaces to help
  students learn about university
• Develop practices which encourage engagement
• Provide students with feedback on their views

57
Manage student engagement

• Shape support systems in response to known
  individual and cohort differences, perceptions
  and reports
• Shape adaptive spaces, portals, services,
  cultures, activities, programs, approaches,
  people
• Study student readiness, identify at risk
  students, studying student transitions through
  study
• Develop seamless links between academic and
  non-academic aspects of university

58
Benchmarking

• Make comparisons within and between institutions
• Develop meaningful points of reference and for
  improvement
• Develop learning communities to generate and
  disseminate key ideas

59
At your institution

• How could student engagement data be better used
  at your institution?
• What changes in data collection, analysis and
  reporting may be required?
• What programs or practices could be altered, and
  how?

60
(No Transcript)