Program Assessment

1
Program Assessment

• Kelly Aune
• Office of the Vice Chancellor for
• Academic Affairs, UH-Manoa

2
Initial Questions

• What have we done to our students as a result of
  their exposure to our program?
• Problem You should know what you did to them,
  or at least what you intended to do to them.
• Did we do to our students what we intended to do
  to them?
• Advantage Requires learning outcomes.
• Problem All or nothing approach.

3
Better Questions

• To what extent have we changed our students in
  the ways we intended to change them?
• Advantage Allows for range and variability of
  expected changes, i.e., more sensitive to change.
• To what extent have we changed our students on X,
  Y, and Z?
• Advantage Sensitive and more specific.

4
Obvious First Step

• If you have not done so yet, sit down as a
  faculty and articulate specific learning outcomes
  or objectives, specific effects your program is
  designed to produce in students.
• Make certain your learning objectives lend
  themselves to measurement. Be a better person
  is tough to measure.

5
Nature of Effects

• Cognitive effects changes in
• Knowledge beliefs
• Understandings
• Skills and Capabilities (e.g., calculate
  introspect evaluate analyze synthesize
  ability to apply)
• Attitudinal effects changes in
• Predispositions preferences opinions.

6
Nature of Effects

• Behavioral effects changes in
• Various written communication skills
• Various oral communication skills
• Social skills
• Organizational skills

7
Assessment

• Cheap assessment -- e.g., paper/pencil surveys
  and testing.
• Advantages low dollar cost minimal labor
  requirements fast efficient.
• Can be used for low-level cognitive data.
• Disadvantages often limited to student
  self-report and/or attitudinal data.

8
Assessment

• Expensive assessment
• Disadvantages can require significant
  investment to cover expenses such as incentives
  and labor costs.
• Advantages can provide data on higher order
  cognitive effects and behavioral performance
  indicators.

9
Cheap Assessment

• Attitudinal surveys
• Instructor evaluations
• Course content evaluations
• Advantages fast easy
• Disadvantages tells us what students
  think/believe about a course or instructor but
  not what the class has done for them.

10
Cheap Assessment

• Self report surveys
• Perceptions of capabilities
• Perceptions of understandings
• Perceptions of experiences e.g., communication
  apprehension
• Advantages fast easy perceptions of
  experiences can be very useful.
• Disadvantages self reports of ones
  capabilities and understandings are always
  suspect.

11
Better Cheap Assessment

• Knowledge based assessment (a.k.a. multiple
  choice test).
• Create a comprehensive test consisting of
  sample questions from all core or required
  courses.
• Administer to selected groups at selected times.
• Advantages fast easy actually provides an
  indication of learning.
• Disadvantages provides a most superficial
  aspect of learning, a narrow bandwidth of
  cognitive change.

12
Expensive Assessment

• Recording/Assessment of behavior
• Recording of speech/presentation for subsequent
  analysis by coders.
• Written responses to stimuli materials for
  subsequent analysis by coders.
• Portfolio collection for subsequent analysis by
  judges/experts.
• Recording of performance for subsequent
  assessment by judges/experts.
• Track your alumni

13
Expensive Assessment

• Advantages provides actual performance
  indicators.
• Disadvantages time intensive and labor
  intensive costs may be generated for incentives
  (for students) pay for coders/judges coders
  need to be trained to ensure reliability and
  validity of assessments.

14
Design

• Single Data Set
• Advantages fast easy yields one set of
  numbers.
• Disadvantages what can that single set of
  numbers tell you? How do you interpret those
  numbers?
• Comparison Data Sets Time 1 vs. Time 2 or
  comparison group.
• Disadvantages more labor intensive analysis
  more complicated.
• Advantages allows comparison of data relative
  to learning stimuli.

15
What To Do With Your Data

• Report Descriptive Statistics
• Advantages fast easy requires little
  expertise to compile.
• Disadvantages how do you argue for
  effectiveness?
• Report Statistical Comparisons
• Advantages allows for strong arguments
  concerning learning outcomes.
• Disadvantages requires some expertise.

16
Closing the Loop

• Interpreting the results
• What do the data tell you about the effectiveness
  of your program?
• Altering your program
• Adjust curriculum content, sequencing of courses,
  prerequisites, etc., in response to results.
• Test Again!

17
Final Thoughts

• Forget WASC
• Yes, really truly forget WASC!
• Do this for yourselves. The only way to establish
  the culture of evidence that WASC is looking
  for is to find value in the assessment process.
  Do it thoughtfully, with scholarly curiosity,
  with an expectation of value and you will find
  this a rewarding process.