Measuring and Managing Student Engagement:

1

• Measuring and Managing Student Engagement
• Why it Matters in the First Year of College
• Jillian Kinzie
• Indiana University Center for Postsecondary
  Research

Wisconsin OIPD Spring Conference 2007 Partnering
for Student Success in the First Year Learning
About Learning
2
Current Concerns about College Student Learning

• Colleges and universities, for all the benefits
  they bring, accomplish far less for their
  students than they should. -- Derek Bok
• At a time when the quality of postsecondary
  education seems to be slipping, participating in
  engaged learning activities promises to prepare
  students for a lifetime of continuous learning so
  that they and the country stay competitive in the
  global marketplace -- George Kuh
• The first year presents a unique opportunity to
  engage students in the habits of learning. --
  John Gardner

3
Students and the College Experience

• College-going stakes higher today than at any
  point in history
• 45 students in 2yr-colleges depart during their
  first year, 1 of 4 leave from 4-yr schools
• 51 of high school grads have reading skills
  necessary for college 25 of students in 4-yr
  colleges need 1 yr of remedial coursework
• Enrollment persistence rates of historically
  underserved students lagging

4
What We Know About the Undergraduate Experience
from NSSE

1. Full-time students spend an average of 13 hrs per
   week studying (Hrs. recommended by faculty
   20-25)
2. 45 of all college seniors took at least one
   course at another institution prior to enrolling
   at their current school swirling
3. Three of ten first-year students reported working
   just hard enough to get by.
4. Between 40 and 50 of first-year students never
   used career planning, financial advising, or
   academic tutoring services.
5. 45 first-years and 30 seniors never discussed
   ideas with faculty outside class
6. Faculty spend 42 of class time lecturing (FSSE)

5
Current Context for Concern

• Engaged pedagogy is really the only type of
  teaching that truly generates excitement in the
  classroom that enables students and professors to
  feel the joy of learning -- bell hooks
• Conclusion The first year experience is not
  being used as effectively as it could to engage
  students at high levels to develop habits of
  learning that insure student success

6
Principles of Good Practice for the First College
Year Recommendations

• Institutional commitment by leaders, faculty,
  staff, and governing boards
• Focus on student learning both inside outside
  the classroom
• Encourage student affairs-academic affairs
  partnership
• Offer challenge and support
• Communicate high expectations
• Foster an inclusive and supportive campus climate
  
• Conduct systematic assessment
• Create an atmosphere of dignity and respect for
  first-year students
• Teach students strategies and skills to succeed
• Get faculty involved
• Encourage students to assume responsibility for
  their success
• Challenging and Supporting the First-Year
  Student A Handbook for Improving the First Year
  of College By M. Lee Upcraft, John N. Gardner,
  Betsy O. Barefoot, and Associates, Jossey-Bass,
  2005 Pg. 515-517

7
To Ponder

• What data do you use to inform your understanding
  of the first year experience? In teaching first
  year students?
• In what curricular and co-curricular areas,
  specific courses, or transition places do first
  year students have the most difficulty? What
  measures do you use to monitor these issues?
• What are your current strengths in the first year
  experience?

8
Student Success in College

• Student success - defined in a broad,
  all-encompassing manner includes academic
  achievement, engagement in educationally
  purposeful activities, satisfaction, acquisition
  of desired knowledge, skills and competencies,
  persistence, attainment of educational objectives
  including graduation, and post-college
  performance.

http//nces.ed.gov/npec/pdf/Kuh_Team_Report.pdf
9
Study Habits Peer Involvement Interaction
with Faculty Time on Task Motivation Other
First Year Experience Academic Support
Campus Environment Time on Task Peer
Support Teaching Learning Approaches
Other
Pre-college experiences
10
What Really Matters in College Student
Engagement

• The greatest impact appears to stem from
  students total level of campus engagement,
  particularly when academic, interpersonal, and
  extracurricular involvements are mutually
  reinforcing

Pascarella Terenzini, How College Affects
Students, 2005, p. 647
11
What Really Matters in College Student
Engagement

• Because individual effort and involvement are
  the critical determinants of college impact,
  institutions should focus on the ways they can
  shape their academic, interpersonal, and
  extracurricular offerings to encourage student
  engagement.

Pascarella Terenzini, How College Affects
Students, 2005, p. 602
12
Student Success Lessons From the Research

• What matters most is what students do, not who
  they are
• A key factor is the quality of effort students
  expend
• Educationally effective institutions channel
  student energy toward the right activities

13
Two Components of Student Engagement

• 1. What students do -- time and energy devoted to
  educationally purposeful activities
• 2. What institutions do -- using effective
  educational practices to induce students to do
  the right things

14
Foundations of Student Engagement

• Time on task (Tyler, 1930s)
• Quality of effort (Pace, 1960-70s)
• Student involvement (Astin, 1984)
• Social, academic integration (Tinto,1987, 1993)
• Good practices in undergraduate education
  (Chickering Gamson, 1987)
• College impact (Pascarella, 1985)
• Student engagement (Kuh, 1991, 2005)

15
Good Practices in Undergraduate Education
(Chickering Gamson, 1987 Pascarella
Terenzini, 2005)

• Student-faculty contact
• Active learning
• Prompt feedback
• Time on task
• High expectations
• Respect for diverse learning styles
• Cooperation among students

16
Lessons from Research on College Impact

• Effective Educational Practices for Fostering
  Engagement
• Academic Challenge
• Active and Collaborative Learning
• Student-Faculty Interaction
• Enriching Educational Experiences
• Supportive Campus Environments

17
Promise ofStudent Engagement
If faculty and administrators use principles of
good practice to arrange the curriculum and other
aspects of the college experience, students
would write more papers, read more books, meet
with faculty and peers, and use information
technology appropriately, all of which would
result in greater gains in such areas as critical
thinking, problem solving, effective
communication, and responsible citizenship.
Kuh, Kinzie, Schuh, Whitt Associates, Student
Success in College, 2005
18
National Survey of Student Engagement(pronounced
nessie)Community College Survey of Student
Engagement(pronounced cessie)

• Undergraduate student surveys that assess the
  extent to which students engage in educational
  practices associated with high levels of learning
  and development

19
NSSE Project Scope

• One million students from 1,100 different
  schools
• Random sample of First-year and Seniors surveyed
  spring term
• 50 states, Puerto Rico, Canada
• 70 consortia WISCONSIN SYSTEM!

• Related Initiatives
• Faculty Survey of Student Engagement (FSSE)
• Beginning College Survey of Student Engagement
  (BCSSE)

20
NSSE sightings all over Wisconsin!
21
Taking a look at NSSE

• Based on effective educational practices
• Designed and tested for high validity and
  reliability
• Relatively stable over time
• High credibility of self-reported data
• Students will participate
• Actionable data
• Psychometric properties document on Web site

22
NSSE Survey
Student Behaviors
Student Learning Development
Institutional Actions Requirements

Reactions to People Environment
Student Background Information
23
NSSE Survey

• Q.1 Academic activities
• Q.2 Learning mental activities
• Q.3 Reading writing
• Q.4 Homework
• Q.5 Academic challenge
• Q.6 Co-curricular activities

• Q.7 Enriching educational experiences
• Q.8 Campus relationship
• Q.9 Time usage
• Q.10 Institutional emphasis
• Q. 11 Gains
• Q.12 14 Satisfactions

24
NSSEs Five Clusters ofEffective Educational
Practice
Enriching Educational Experiences
Supportive Campus Environment
Level of Academic Challenge
Active Collaborative Learning
Student Faculty Interaction
25
NSSE Results

• A window into the undergraduate experience
• Discover strengths and weaknesses in educational
  program
• Identify areas that need attention to improve
  student learning and success
• Help pinpoint aspects not in line with mission,
  or what institution expects
• Link with other institutional data

26

• What can we learn about first year student
  engagement from NSSE?

27
What do first-year students do?
1. What percent of full-time first-year students
study, on average, more than 20 hours per week?
(a) 12 (b) 18 (c) 30 (d) 41 (e) 50
b. 18 NSSE FY Wisconsin FY 16
28
Worrisome Gap? Time spent studying

• First-year students average about 13-14 hrs. per
  week studying
• Faculty Survey of Student Engagement (FSSE) data
  indicate that faculty expect students to spend
  more than twice that amount preparing (estimated
  24-30 hrs. a week for FT)
• Entering first-year students EXPECT to study more
  than they actually do in college

29
What do first-year students do?
2. What percent of first-year students frequently
(very often often) asked questions in
class or contributed to class discussions?
(a) 28 (b) 35 (c) 47 (d) 58 (e) 65
d. 58 NSSE FY Wisconsin 51 WI Range 31 -
57
30
First Year Students Ask Questions in Class
31
Active and Collaborative Learning Scales

• Collaborative Learning Scale
• Worked with peers on projects during class
• Worked with classmates outside of class to
  prepare assignments
• Tutored/Taught other students
• Discussed ideas from readings or classes with
  others outside of class (students, family)

• Active Learning Scale
• Asked questions in class
• Make a class presentation
• Participated in a community-based project
  (service learning) as part of a regular course

32
What do first-year students do?
3. True or False. Almost half of all students
spend no time on co-curricular activities.
True. 43 NSSE first-years, 48 seniors
report 0 hours
33
FY Students Time Spent on Cocurriculars
34
What do first-year students do?
4. What percent of first year students report
they frequently (often or very often) received
prompt feedback on their academic
performance? (a) 27 (b) 35 (c) 44 (d) 53 (e)
none of the above
d. 53 NSSE first-years, Wisconsin 48
35
FSSE NSSE comparison Prompt Feedback
Lower Division
Upper Division
FACULTY gave prompt feedback often or very often
93 93
1st yr. Students
Seniors
STUDENTS received prompt feedback often or very
often
64 76
36
What do first-year students do?

• 5. What percent of first-years frequently
  prepared 2 drafts of a paper or assignment
  before turning it in?
• ( very often and often)
• (a) 65 (b) 56 (c) 43 (d) 31
• b. 56 NSSE FY Wisconsin 55

37
Striving for Deep Learning

• Deep learning is learning that takes root in
  our apparatus of understanding, in the embedded
  meanings that define us and that we use to define
  the world.
• J. Tagg (2003). The learning paradigm college
  (p. 70). Bolton, MA Anker

38
Deep Learning Sub-ScaleIntegrative Learning

• Worked on a paper or project that required
  integrating ideas or information from various
  sources
• Included diverse perspectives (different races,
  religions, genders, political beliefs, etc.) in
  class discussions or writing assignments
• Put together ideas or concepts from different
  courses when completing assignments or during
  class discussions
• Discussed ideas from your readings or classes
  with faculty members outside of class
• Discussed ideas from your readings or classes
  with others outside of class (students, family
  members, co-workers, etc.)

39
Deep Learning ItemIntegrative Learning Scale

• Students indicate 1 never to 4 very
  often, they did the following during school
  year very often and often
• Worked on paper/project that required integrating
  ideas, info. from various sources
• Included diverse perspectives (different races,
  religions, genders, political beliefs, etc.) in
  class discussions or writing assignments
• Put together ideas or concepts from different
  courses when completing assignments or during
  class discussions
• Discussed ideas from readings or classes with
  faculty members outside of class
• Discussed ideas from readings or classes with
  others outside of class (students, family
  members, co-workers, etc.)

NSSEville U FY - SR
85/92 /
79/82 /
61/80 /
25/40 - /
45/60 - /
40
Compensatory Effect of Engagement

• - Student engagement positively related to FY
  and senior student grades and to persistence
  between the first and second year of college at
  the same institution
• - Engagement has compensatory effect on FY
  grades and persistence to the second year of
  college.

41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
Using NSSE results in your explorations of first
year student learning

• Take a look at the instrument. What items or
  scales might you want to know how your first year
  students score?
• - what comparison group (to seniors, peers at
  other similar institutions, among departments,
  criterion reference) is compelling?
• What items do you want to know more about what
  students responses really mean?
• What does prompt feedback mean to students?
• What assignments do students think require them
  to synthesize ideas?
• How else might you assess the effective
  educational practices on NSSE?
• How might you use NSSE results to warrant or add
  to an inquiry project of interest to you?

45
Approaches to using NSSE in inquiry about
teaching and learning

• Assess engagement in your class.
• Add selected NSSE items to your course evaluation
  to obtain classroom level engagement data,
  compare to aggregate FY data
• Use NSSE to pinpoint FY results deserving of
  additional inquiry
• Ex FY student faculty engagement low. Conduct
  student faculty focus groups to discover
  obstacles to interaction
• Examine group differences
• Identify FY groups of interest (e.g., rural
  students, first-generation, majors, women in
  science) -examine difference
• Assess impact of intervention
• Ex Effect of FY seminar on engagement impact of
  service learning course on reflective learning
  and civic engagement outcomes

46
Localizing Findings

• Variation in levels of student engagement within
  the university is greater than variation between
  universities
• Improvement initiatives might best be designed
  and implemented at the college level (rather than
  based on institutional findings) to maximally
  impact overall student engagement on campus.

47
Localizing Findings contd

• Even when institutional outcomes are
  disaggregated to the college levelless desirable
  engagement outcomes are attributed to
  shortcomings in another academic department
• These less-engaged students are not my students
  
• Many institutions have increased their
  institutional NSSE sample or done targeted,
  over-sampling efforts to specific majors.

48
Localizing Findings contd

• Disaggregating NSSE outcomes to the department
  level often leads to the same series of
  observations that occurred at the college level
• a faculty member within the department will often
  explain that these less-than-optimal levels of
  engagement are not characteristic of students in
  my classes, but rather must be in other course
  offerings within the department. These are not my
  students!

49
CLASSE (Classroom Survey of Student Engagement)
created by Bob Smallwood, Univ. North Florida and
Judy Ouimet, Univ. of Nevada, Reno

• CLASSE is a pair of survey instruments that
  enable one to compare what engagement practices
  faculty particularly value and perceive important
  in a designated class with how frequently
  students report these practices occurring in that
  class.
• CLASSEStudent is the survey instrument completed
  by each student enrolled in the designated class,
  while CLASSEFaculty is the survey instrument
  completed by the faculty instructor of the
  designated class.

50
Approaches to using NSSE in inquiry about
teaching and learning

• Assess engagement in your class.
• Add selected NSSE items to your course evaluation
  to obtain classroom level engagement data,
  compare to aggregate FY data
• Use NSSE to pinpoint FY results deserving of
  additional inquiry
• Ex FY student faculty engagement low. Conduct
  student faculty focus groups to discover
  obstacles to interaction
• Examine group differences
• Identify FY groups of interest (e.g., rural
  students, first-generation, majors, women in
  science) -examine difference
• Assess impact of intervention
• Ex Effect of FY seminar on engagement impact of
  service learning course on reflective learning
  and civic engagement outcomes

51
Study effective practices

• Teaching Learning Center at University of
  Wisconsin-Stout interviewed students to develop a
  more contextualized understanding of student
  engagement
• TLC staff distilled a list of effective
  educational practices from the interview data,
  particularly around the most important factor to
  students student-faculty relationships.

52
Focus on desired pedagogy

• First-year students less involved in service
  learning than JMU desired.
• Workshops conducted to encourage faculty to adapt
  courses to include service learning
• Studied change in participation of students and
  instructional practice

53
Improving teaching learning

• NSSE and CIRP pointed to problems with first year
  students academic engagement, but WTAMU desired
  more holistic picture of students experience
• Conducted Student Engagement Audit Focus Groups
  2 focus groups per college to discover what
  faculty and students found educationally engaging
  and identify classroom experiences that were
  engaging and disengaging

54
Approaches to using NSSE in inquiry about
teaching and learning

• Assess engagement in your class.
• Add selected NSSE items to your course evaluation
  to obtain classroom level engagement data,
  compare to aggregate FY data
• Use NSSE to pinpoint FY results deserving of
  additional inquiry
• Ex FY student faculty engagement low. Conduct
  student faculty focus groups to discover
  obstacles to interaction
• Examine group differences
• Identify FY groups of interest (e.g., rural
  students, first-generation, majors, women in
  science) -examine difference
• Assess impact of intervention
• Ex Effect of FY seminar on engagement impact of
  service learning course on reflective learning
  and civic engagement outcomes

55
Gender Differences in Engagement by First-year
Students (source NSSE 2006)
Female Students More Engaged Male Students More Engaged
Community service or volunteer work Tutored or taught other students
Hours per week preparing for class Hours per week relaxing and socializing (watching TV, partying, etc.)
Hours per week providing care for dependents living with you (parents, children, spouse, etc.) Hours per week participating in co-curricular activities
Worked harder than you thought you could to meet an instructor's standards or expectations Exercised or participated in physical fitness activities
Used e-mail to communicate with an instructor Discussed ideas from readings or classes with faculty members outside of class
Foreign language coursework Number of problem sets (problem-based homework assignments) that take less than an hour to complete
Prepared two or more drafts of a paper or assignment before turning it in Research project with faculty member
Attended an art exhibit, gallery, play, dance, or other theater performance Worked with faculty members on activities other than coursework (committees, orientation, student life activities, etc.)
56
Approaches to using NSSE in inquiry about
teaching and learning

• Assess engagement in your class.
• Add selected NSSE items to your course evaluation
  to obtain classroom level engagement data,
  compare to aggregate FY data
• Use NSSE to pinpoint FY results deserving of
  additional inquiry
• Ex FY student faculty engagement low. Conduct
  student faculty focus groups to discover
  obstacles to interaction
• Examine group differences
• Identify FY groups of interest (e.g., rural
  students, first-generation, majors, women in
  science) -examine difference
• Assess impact of intervention
• Ex Effect of FY seminar on engagement impact of
  service learning course on reflective learning
  and civic engagement outcomes

57
Study effects of potentially High Impact
Practices

• First-year seminars and experiences
• Common intellectual experiences
• Learning communities
• Writing-intensive courses
• Collaborative assignments
• Science as science is done
• Diversity/global learning
• Service learning/community service
• Internships
• Capstone experiences/projects

58
UW Green Bay Fall 2006 Freshman Seminar Pilot
Study

• Comparing Freshman Seminar students from
  non-Seminar students
• Asked questions in class or contributed to class
  discussions.
• Worked with other students on projects DURING
  CLASS.
• Used e-mail to communicate with an instructor.
• Talked about career plans with a faculty member
  or advisor.
• Discussed grades or assignments with an
  instructor.
• Coursework emphasized APPLYING theories or
  concepts to practical problems or in new
  situations.
• Coursework emphasized MAKING JUDGMENTS about the
  value of information, arguments, or methods, such
  as examining how others gathered and interpreted
  data and assessing the soundness of their
  conclusions

59
Institutional ExamplesExploring the First Year
Experience
60
California State University,

• Seven principles of Good Practice in
  Undergraduate Education
• NSSE results indicated need to focus on
  first-year experience
• Campus-wide task force used selected NSSE data to
  determine how well students were doing and to
  pinpoint areas that needed improvement
• First step - philosophy of First Year Experience
  distributed to entire campus

61
Plymouth State University

• Student Affairs reviews NSSE data to assess
  out-of-class support
• NSSE and institutional survey data used to revise
  General Education program
• NSSE results supported grant application for
  faculty development center

PSU
62
St. Olaf College

• Committee on First-Year Experience (CoFYE),
  first-year curriculum faculty student life
  staff conducted self-study, used NSSE results to
  compare St. Olaf FY to peers on key indicators,
  conducted focus groups with students and faculty
  using questions similar to NSSE items to gain
  insight into their results.
• Self-study report serves as baseline for
  assessing change and to launch revisions to
  First-Year Experience

63
Truman State University

• NSSE results showed need for

improvement in academic advising

• Implemented professional advisors in residence
  halls
• Based on NSSE and internal survey, 50 faculty
  were interested in advising workshop
• Recommendation to provide faculty with regular
  workshops and advising newsletter
• VP of Academic Affairs to carry out comprehensive
  assessment of advising on campus

64
University of Akron

• Distribute campus-wide philosophy
• More emphasis on diversity in FY and Gen Ed.
  Courses
• Reward system for faculty teaching FY courses
• One Voice syllabi common components in FY
  curriculum
• Promote service learning, mentoring,
  undergraduate research
• Foster student-faculty interaction
• Student Success Seminar

65
Worcester Polytechnic Institute

• NSSE results showed FY students were less engaged
  than seniors
• New FY interdisciplinary, inquiry-based seminars
  better integration of disciplines engaging
  introductory courses
• Associate Dean appointed to Office for the First
  Year
• Assessment plan in development with NSSE
  indicators as key component

66
FSSE Survey

• Faculty perceptions of how often their students
  engage in different activities
• The importance faculty place on various areas of
  learning and development
• The nature and frequency of interactions faculty
  have with students
• How faculty members organize class time

67
FSSE- NSSE inquiry Teaching and Learning

• Use FSSE to explore pedagogical practices by
  school, department, lower division vs. upper
  division faculty
• Compare NSSE and FSSE to examine gaps between
  faculty and student perceptions and behaviors

68
FSSE - Class time
69
NSSE FSSE Time Spent Preparing for Class
70
Beginning College Student Survey of Engagement
(BCSSE)

• BCSSE measures
• HS experiences Importance of college activities
• Baseline self-assessment of knowledge and skills
• Demographics
• Use to examine gap between expectations and
  engagement
• Study effect of students background on NSSE
  scores

71
New Student Expectations, Engagement and Success
in College

• Path to student success begins pre-college,
  shaped by early experiences with institution
• Overall pattern Students initial expectations
  for their academic involvement is less than
  faculty might wish, and their actual engagement
  is even less
• Students set in place in their first semester of
  college the pattern of time allocation that will
  serve them across their years in college
• Educators need to detail the specific activities
  that will lead to success in college

72
QA

• For More Information
• Email jikinzie_at_indiana.edu
• NSSE Web site www.nsse.iub.edu