Using Measurable Outcomes to Evaluate Tutor Programs

1
Using Measurable Outcomes to Evaluate Tutor
Programs

• Jan Norton,
• Presenter

2
Introduction

• Jan Norton
• Email snowfanjan_at_gmail.com
• Phone [phone]
• Masters in Educational Research Psychology
• 25 years as director of multi-discipline
  tutoring centers with CRLA certifications
• Former reviewer for CRLA tutor training
  certification current NADE certification
  reviewer for tutoring, SI, and developmental
  education programs
• NCLCA Lifetime Learning Center Leadership
  certification

3
Webinar Contents

• Terminology
• The Black Box
• Timetables
• Data/Measurement
• Data/Measurement Cautions
• Program Outcomes Examples
• Student Outcomes Examples
• Closing

4
Terminology

• Visions
• Missions
• Values
• Goals
• Objectives
• Outcomes
• Program vs. Student Outcomes
• Learning Outcomes

5
The Black Box
Something Happens
6
The Black Box
Something Happens
What is going on in there? How do we measure
it?
7
Timetables

• Scope of assessment
• 6 years
• 2 years
• 1 year
• 1 semester
• 1 course
• 1 assignment
• 1 tutoring session

8
Timetables

• Planning for Assessment
• Continual processes
• Multiple times per semester
• Once or twice per semester
• Two or three times per year
• Once per year
• Every two years
• Every 4 years

9
Data/Measurement

• Why Measure
• What to measure
• Before (Benchmarks)
• During
• After (Outcomes)

10
Data/Measurement

• How to measure
• Program-generated data
• Course-specific data
• Campus data
• Campus surveys
• Standardized tools
• Locally developed tools

11
Data/Measurement

• More about locally developed tools
• Surveys
• Likert-type scales
• Explicated scales
• Behavioral rubrics
• Self assessment by student
• Tutor assessment
• Observer assessment

12
Data/Measurement Cautions

• Direct vs. Indirect Measures
• Data access and consistency
• Realistic limitations
• Cooperation
• The Baseline Challenge
• Time money
• (and staff and comfort and )

13
Chat Time for Questions Contributions

• Finished the foundation what we can and cant
  measure, when how to measure, and what data we
  will gain
• Moving on to examples of program-level outcomes

14
Program Outcomes

• Possible Areas of Assessment Focus
• Establishing baselines
• Usage
• Satisfaction or quality
• Level/variety of services
• Staff development
• Reaching certification or meeting standards

15
Program Outcomes Examples

• Outcome Statement
• As a result of the new marketing strategy, the
  number of students attending group tutoring
  sessions will increase by 10 in Spring 2013.
• Timetable
• One semester
• Starting Measure (Benchmark)
• Average of past attendance
• Ending Measure
• Workshop attendance records

16
Program Outcomes Examples

• Outcome Statement
• During 2012-13, the number of tutors who earn
  CRLA Level I certification will increase by 5.
• Timetable
• One academic year
• Starting Measure (Benchmark)
• Number of tutors achieving Level I in previous
  year(s)
• Ending Measure
• Number of tutors achieving Level I in 2012-13

17
Program Outcomes Examples

• Outcome Statement
• Outside reviewers using the NADE self-study
  process will score the tutoring program at 85 or
  above on all sections.
• Timetable
• One year
• Starting Measure (Benchmark)
• Internal review self-study
• Ending Measure
• External review

18
Program Outcomes Examples

• Outcome Statement
• As a result of tutor training sessions, tutors
  will demonstrate an improved understanding of
  Blooms Taxonomy.
• Timetable
• One semester or year, depending on training
  schedule
• Starting Measure (Benchmark)
• Pre-test
• Ending Measure
• Post-test

19
Chat Time for Questions Contributions

• Finished program-level outcomes
• Moving on to examples of student outcomes

20
Student Outcomes Measures

• Qualitative
• Interviews focus groups
• Opinion surveys
• Observations or self reports
• Quantitative
• Assignment course grades
• Instrument scores
• Observation rubrics

21
Student Outcomes Types

• Affective
• Build confidence
• Improve attitude
• Reduce anxiety
• Cognitive
• Mastery of material
• Conscious strategies use

22
Student Outcomes Examples

• Outcome Statement
• After being tutored for at least one hour,
  clients will be more confident in their ability
  to accomplish the academic task they focused on
  in tutoring.
• Timetable
• One tutoring session
• Starting Measure (Benchmark)
• Client self-report
• Ending Measure
• Client self report

23
Student Outcomes Examples

• Outcome Statement
• Students who work with the study skills tutors
  will demonstrate an increased ability to use
  memorization strategies.
• Timetable
• One tutoring session
• Starting Measure (Benchmark)
• Tutor or observer checklist
• Ending Measure
• Tutor or observer checklist

24
Student Outcomes Examples

• Outcome Statement
• 75 of the students who work with a reading tutor
  for at least two sessions will score at least 75
  on the main ideas reading test.
• Timetable
• Within three weeks
• Starting Measure (Benchmark)
• Reading quiz at start of tutoring
• Ending Measure
• Chapter reading test in class

25
Student Outcomes Examples

• Outcome Statement
• Students who attend at least 10 group tutoring
  sessions for Algebra Review 090 will earn an
  average grade of B or above in the course.
• Timetable
• One semester, length of course
• Starting Measure (Benchmark)
• Placement score into course
• Ending Measure
• Course grade

26
Student Outcomes Examples

• Outcome Statement
• After attending mandatory tutoring for all 5 of
  their developmental education courses, students
  will be retained until the start of their junior
  year at a rate at least 25 higher than
  developmental students for whom tutoring was not
  required.
• Timetable
• Two years
• Starting Measure (Benchmark)
• Developmental course placement
• Ending Measure
• Enrollment level

27
Student Outcomes Examples

• Outcome Statement
• Students attending SI for World Religions will
  apply previously understood information,
  concepts, and experiences to a new situation or
  setting. (CAS)
• Timetable
• Within one semester
• Starting Measure (Benchmark)
• Pre-survey
• Ending Measure
• Post-survey

28
Closing

• Check with your institutional research or
  assessment office
• Review early drafts of outcomes statements with
  your supervisor
• Pre-test any locally created measurement tool
• Be prepared for outcomes that are not met or
  wildly exceeded

29
Closing

• Remember the black box we do not own or control
  the box, and our ability to see into it is very
  limited
• Questions Contributions?

30

• Thank You for Participating !
• Jan Norton
• 
  snowfanjan_at_gmail.com
• [phone]