Assessing Learning Outcomes

1
Assessing Learning Outcomes
Chris Weisbrook August 13, 2009
2
Lets start with a quiz True or
False?Assessment isconfusing.
3
The goal of this presentation is to unconfuse you.
4
And to give you HOPE.
5
What Has Been Done

• Establish campus learning outcomes.
• Determine timeline.

6
What Departments Need to Do By June 2011

• Develop assessment plan for programs.
• Assess learning outcomes.
• Evaluate assessment data.
• Make improvements based on data.
• Assess effect of improvements (close the loop).
• Submit results.

7
But how can I get all of this done in time?!!
8
Im glad you asked that question!

• It is NOT as difficult as you may think.

9
I will show you how!
10
The good news is that youre doing most of the
work of assessment already.

• The key is how to organize and tweak what you are
  already doing.

11
Definitions of Terms Used in This Presentation

• Learning outcomes
• Assessment
• Direct versus indirect assessment
• Evaluation
• Assessment instrument or tool

12
Definition of Learning Outcomes

• Learning outcomes are statements that describe
  what students are expected to know and be able to
  do by the time of graduation. These relate to
  the skills, knowledge, and behaviors that
  students acquire while in the program.
• HLC requires that we assess learning outcomes.

13
Learning Outcomes

• Seven learning outcomes have been established for
  the campus
• Communication
• Critical thinking
• Problem solving
• Diverse environments/teamwork
• Ethical responsibility
• Contemporary issues
• Life-long learning

14
Definition of Assessment

• Assessment involves getting the data.
• Assessment describes one or more processes that
  identify, collect, and prepare data to evaluate
  the achievement of learning outcomes.

15
Direct Assessment Versus Indirect Assessment

• Direct assessment of learning outcomes is
  expected by HLC. Indirect assessment data may
  supplement direct assessment data.
• Direct assessment involves actual observation or
  measurement of student performance using
  assignments, projects, tests, presentations,
  classroom activities, etc.
• Indirect assessment involves reflection upon and
  opinion of a students abilities or achievement
  by the student or others using instruments such
  as surveys and interviews.

16
Definition of Evaluation

• Evaluation describes what is done with the data.
• Evaluation
• involves interpreting the data and evidence
  accumulated through assessment practices.
• results in decisions and actions to improve the
  program.

17
Assessment Tools and Methods

• Assessment tool or instrument is what is used for
  measurement
• Direct assessment
• Assignment, test question, project, report,
  portfolio, simulation, presentation, etc.
• Indirect assessment
• Survey, focus group or interview.

18
What Data Should We Get and How Do We Get It?

• Programs may use whatever method they choose as
  long as they get direct assessment data.
• However, if you are flummoxed about how to do
  this, you may want to adopt a procedure that a
  number of programs are using that is efficient,
  effective and easy to comprehend.

19
The Procedure

• Since each student must be given an opportunity
  to meet every outcome, the cleanest approach is
  to start with the required courses in the
  curriculum.
• Map the learning outcomes to the courses to
  determine which courses support which outcomes.
• Thulasi has prepared a table of required courses
  versus outcomes for some of your programs.
• The task is to come up with an plan to assess the
  outcomes in those courses.

20
Huge Caution More is not better!

• Assessment must be strategic and sustainable.
• Goal is to assess each learning outcome at least
  once and preferably three to five times during a
  students academic career.

21
Guidelines for Choosing Assessment Instruments

• Assessment instrument must be focused on the
  outcome or on one component of the outcome.
• as simple as a single question on a test or
• as complex as a 10-page term paper.
• Test grades and course grades are generally not
  appropriate because they represent aggregate
  average performance over many areas of skill and
  knowledge.

22
How should assessment instruments be chosen?
23
Start with the basic question
Why assess learning outcomes?
Answer
To obtain information useful for program
improvement.
24
Program Improvement

• To improve program, assessment data must be at a
  degree of specificity so that changes can be made
  that focus on correcting a shortcoming.
• Learning outcomes are fairly broad.
• It is advisable to break learning outcomes into
  specific components that are measureable.

25
Program Improvement

• For example, how would one address shortcomings
  in the communications outcome unless the specific
  areas of weakness were known?
• Shortcoming could be in oral communication,
  general written communication, technical reports,
  or another area.

26
Performance Criteria for Assessment

• Specific components of an outcomes are known as
  performance criteria.
• Performance criteria are narrower and more
  specific statements and therefore they are easier
  to measure.

27
Developing Performance Criteria

• Develop performance criteria by considering the
  essential goals of your program with respect to
  each of the learning outcomes.
• Use an action verb to describe the goal.
• For initial assessment cycle, write at least one
  and preferably two or three performance criteria
  for each outcome.

28
Example of Performance Criteria(Pick one or more
of the examples or write your own)

• Learning outcome CommunicationProgram must
  demonstrate that their graduates have an ability
  to communicate effectively both orally and in
  writing.
• Performance Criteria (examples)
• Communicates effectively in individual oral
  presentations.
• Communicates effectively in group oral
  presentations.
• Communicates effectively in written technical
  reports and memos.
• Communicates effectively in written non-technical
  reports.
• Communicates effectively in email communications.

29
Assessment Instruments for Performance Criteria

• In general, the performance criteria describe
  what you are already doing in your program.
• Use the table of required courses versus learning
  outcomes to match each performance criterion to a
  course for assessment.
• Course instructors select an assessment
  instrument (assignment, research paper, test
  question, etc.) to use for each performance
  criterion matched to that course.

30
Assessing Performance Criteria

• A numerical score such as the percentage of
  problems correct on an exam or portion of an exam
  may be appropriate for applies knowledge of
  statistics.
• A rubric would be a more appropriate method to
  use for communicates effectively in group oral
  presentations.

31
Developing Rubrics

• Steps
• Break performance criteria into categories.
• Describe levels of achievement for each category
  based on expectationstypically 3 or 4 levels
• See example

32
(No Transcript)
33
Assessment Using Rubrics

• Data is recorded in a table and consolidated for
  the section or the course.
• For the example shown, the instructor would
  indicate the level of achievement in each of the
  categories during the presentation.

34
Metrics for Performance Criteria

• Metrics define the level of accomplishment
  required for a performance criterion to be
  considered achieved or met for your program.
• Examples
• Using rubric Performance criterion is met if 75
  of students meet or exceed expectations in all
  categories of the performance criterion.
• Using score Performance criterion is met if 75
  of students receive 8/10 or better on test
  question 4.

35
Metrics for Outcomes

• Thresholds must also be set for learning outcomes
  in terms of results on performance criteria.
• Examples
• Learning outcome is satisfied if all performance
  criteria for that outcome are met.
• Learning outcome is satisfied if two of the three
  performance criteria for that outcome are met.

36
Evaluation at Program Level

• Consolidate results for each learning outcome.
• Determine if learning outcome is met.
• Use results of assessment to determine if changes
  need to be made in curriculum, courses,
  prerequisites, performance criteria or metrics.
• Implement change.
• Assess again to determine effect of change.

37
Outcomes Assessment Schedule

• In general
• Not every student has to be assessed for every
  performance criteriona sampling is fine as long
  as the sample is representative of the entire
  population.
• Not every learning outcome or performance
  criterion has to be assessed every semester.

38
However, we need data fast!

• So heres the schedule
• Fall 2009 Develop assessment plan for your
  programs.
• Write at least one, and preferably two or three,
  performance criteria for each outcome.
• Identify a course in which to assess each
  performance criterion.
• Develop rubric or other means of assessment.
• Set metrics for meeting performance criteria and
  for meeting outcomes.
• Spring 2010 Where possible, assess performance
  criteria.
• May 2010 (before faculty leave for summer)
  Evaluate data in terms of meeting performance
  criteria and outcomes, and determine response for
  improvement.
• Fall 2010 Implement changes, where possible, and
  assess again. Assess performance criteria for
  courses not taught in spring.
• Spring 2011 Evaluate data and close the loop,
  where possible (i.e., determine effect of
  changes). Submit results for programs.

39

• Questions?