Assessment 101: The Core Curriculum

1
Assessment 101The Core Curriculum

• Susan Hatfield - Winona State
• UNC Wilmington
• November 2005
• SHatfield_at_winona.edu

2
Common Reactions to Assessment Initiatives

• Ignoring it
• Bribing someone else to do it
• Complaining about it
• Losing sleep over it
• Sitting down and writing it

3

• Assessment is
• -- first and foremost -- about student learning.

4
A Little Quiz
5
Quiz

• All faculty need to be actively engaged in
  assessment for a department or program to really
  be doing assessment.

6
Quiz

• All faculty need to be deeply committed to
  assessment for a department or program to really
  be doing assessment.

7
Attitudes toward Assessment
70
15
15
Hostile
Accepting
Enthusiastic

• Level of Commitment

8
Quiz

• The best way to build a campus culture of
  assessment is for top administration to be
  prescriptive in student learning outcomes,
  assessment measures and methods.

9
Quiz

• All departments and programs should be moving
  ahead on assessment at the same pace, meeting
  specific goals within specified time frames.

10
Quiz

• Effective assessment programs have eliminated
  indirect measures and measures of departmental
  effectiveness (satisfaction, etc.).

11
Quiz

• Assessment is a time-intensive add-on that will
  be a huge burden to faculty who are already
  overburdened.

12
Quiz

• Effective programmatic assessment plans have
  every faculty member assess every outcome in
  every course every semester.

13
Quiz

• Accrediting organizations expect to see fully
  realized assessment plans during site visits.

14
Quiz

• The most effective assessment programs are ones
  in which the students are not aware they are
  being assessed.

15
Uncertainty

• Uncertainty related to questions of
• How to do it
• Why it needs to be done
• What to do with the data
• How the data will be used
• How to find the time to implement it
• What support is available

16
Reducing Uncertainty

• Clear
• Understanding
• Definitions
• Processes
• Rules
• Resources
• Understanding of pitfalls

17
The Assessment Core Curriculum10 things you
should know about assessment
18
Core Curriculum

• Assessment terminology
• Evolution of assessment initiatives
• Difference between direct and indirect measures
  of learning
• Assessment requires exertion and intention
• 5. Writing student learning outcomes

19
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  OUTCOME
• C. Activities and Assignments to help students
  learn LEARNING EVENTS
• D. The key elements related to the accomplishment
  of the outcome COMPONENTS

20
Language of Assessment

• E. The objects of analysis OBJECTS
• F. Data indicating degree of achievement
  CHARACTERISTICS
• G. Combination of data indicating relative
  degree of achievement of the learning outcome
  INDICATORS

21
Goals
22
Goals

• Organizing Principle
• Category or Topic Area
• Subjects

23
Goals

• Composition
• PE
• Humanities
• Fine Arts
• Natural Sciences and Mathematical Sciences
• Social and Behavioral Sciences
• Interdisciplinary Perspectives

24
Learning Outcomes
25
Student Learning Outcomes
Communication
Relating
Speaking
Listening
Teaming
Writing
26
Learning Events
27
Learning Events

• Assignments (in class and out of class)
• Feedback on practice
• Self evaluation
• Peer evaluation
• Role Play
• Pre Tests
• Simulation

28
Learning Objects
29
Student Learning Outcomes
Communication
Relating
Speaking
Listening
Teaming
Writing
Sales
30
Components
31
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
Learning events
Object
32
Student Learning Outcomes
Communication
Relating
Speaking
Listening
Teaming
Writing
Verbal
Demonstration
Evaluative elements
Nonverbal
Organization
33
Performance Characteristics
34
Student Learning Outcomes
Communication
Relating
Speaking
Listening
Teaming
Writing
Verbal
Demonstration
Nonverbal
Organization
35
Indicators
36
Student Learning Outcomes
GOAL
Outcome
Outcome
Outcome
Outcome
Outcome
Degree to which outcome is achieved
component
Object
component
indicator
component
component
37
Core Curriculum

• Assessment terminology
• Evolution of assessment initiatives
• Difference between direct and indirect measures
  of learning
• Assessment requires exertion and intention
• 5. Writing student learning outcomes

38
Maturing Assessment
BEGINNING
PROGRESS
MATURING
INSTITUTIONAL RESPONSIBILITY
DEPARTMENT RESPONSIBILITY
39
Maturing Assessment
BEGINNING
PROGRESS
MATURING
INDIRECT MEASURES
DIRECT MEASURES
40
Maturing Assessment
BEGINNING
PROGRESS
MATURING
PROCESS MEASURES
OUTCOME MEASURES
41
Maturing Assessment
BEGINNING
PROGRESS
MATURING
CLASSROOM ASSESSMENT
PROGRAM ASSESSMENT
42
Maturing Assessment
BEGINNING
PROGRESS
MATURING
INSTITUTIONAL EFFECTIVENESS
STUDENT LEARNING
43
Evolutionary Trajectories
04
05
99
00
01
02
03
98
MATURING
MAKINGPROGRESS
BEGINNING
44
Core Curriculum

• Assessment terminology
• Evolution of assessment initiatives
• Difference between direct and indirect measures
  of learning
• Assessment requires exertion and intention
• 5. Writing student learning outcomes

45
Direct Measures of Learning

• Capstone experience
• Standardized tests
• Performance on national licensure certification
  or professional exams
• Locally developed tests
• Essay questions blind scored by faculty
• Juried review of senior projects
• Externally reviewed exhibitions performances
• Evaluation of internships based upon program
  learning outcomes

46
Indirect Measures of Learning

• Alumni, employer, and student surveys (including
  satisfaction surveys)
• Exit interviews of graduates and focus groups
  graduate follow up studies
• Retention and transfer studies
• Length of time to degree
• ACT scores
• Graduation and transfer rates
• Job placement rates

47
Non-Measures of Student Learning

• Curriculum review reports
• Program review reports from external evaluators
• Faculty publications and recognition
• Course enrollments and course profiles
• Faculty / student ratios, percentage of students
  who study abroad
• Enrollment trends
• 5 year graduation rates
• Diversity of the student body

48
Core Curriculum

• Assessment terminology
• Evolution of assessment initiatives
• Difference between direct and indirect measures
  of learning
• Assessment requires exertion and intention
• 5. Writing student learning outcomes

49
Exertion without Intention
50
Intention without Exertion
51
Intention and Exertion
52
Core Curriculum

• Assessment terminology
• Evolution of assessment initiatives
• Difference between direct and indirect measures
  of learning
• Assessment requires exertion and intention
• 5. Writing student learning outcomes

53
Learning Outcome Format

• Students should be able to
• ltltaction verbgtgt ltltsomethinggtgt

54
COMPREHENSION
EVALUATION
APPLICATION
ANALYSIS
SYNTHESIS
KNOWLEDGE
Associate Classify Compare Compute Contrast Differ
entiate Discuss Distinguish Estimate Explain Expre
ss Extrapolate Interpolate Locate Predict Report R
estate Review Tell Translate
Analyze Appraise Calculate Categorize Classify Com
pare Debate Diagram Differentiate Distinguish Exam
ine Experiment Identify Inspect Inventory Question
Separate Summarize Test
Arrange Assemble Collect Compose Construct Create
Design Formulate Integrate Manage Organize Plan Pr
epare Prescribe ProducePropose Specify Synthesize
Write
Appraise Assess Choose Compare Criticize Determine
Estimate Evaluate Grade Judge Measure Rank Rate R
ecommend Revise Score Select Standardize Test Vali
date
Cite Count Define Draw Identify List Name Point Qu
ote Read Recite Record Repeat Select State Tabulat
e Tell Trace Underline
Apply Calculate Classify Demonstrate Determine Dra
matize Employ Examine Illustrate Interpret Locate
Operate Order Practice Report Restructure Schedule
Sketch Solve Translate Use Write
Lower division course outcomes
55
COMPREHENSION
EVALUATION
APPLICATION
ANALYSIS
SYNTHESIS
KNOWLEDGE
Associate Classify Compare Compute Contrast Differ
entiate Discuss Distinguish Estimate Explain Expre
ss Extrapolate Interpolate Locate Predict Report R
estate Review Tell Translate
Analyze Appraise Calculate Categorize Classify Com
pare Debate Diagram Differentiate Distinguish Exam
ine Experiment Identify Inspect Inventory Question
Separate Summarize Test
Arrange Assemble Collect Compose Construct Create
Design Formulate Integrate Manage Organize Plan Pr
epare Prescribe ProducePropose Specify Synthesize
Write
Appraise Assess Choose Compare Criticize Determine
Estimate Evaluate Grade Judge Measure Rank Rate R
ecommend Revise Score Select Standardize Test Vali
date
Cite Count Define Draw Identify List Name Point Qu
ote Read Recite Record Repeat Select State Tabulat
e Tell Trace Underline
Apply Calculate Classify Demonstrate Determine Dra
matize Employ Examine Illustrate Interpret Locate
Operate Order Practice Report Restructure Schedule
Sketch Solve Translate Use Write
Upper division Course / Program outcomes
56
Learning Outcome Rules

• Only one action verb
• Identify single accomplishments
• Focus on students, not faculty or curriculum

57
Example 1

• Gather factual information and apply it to a
  given problem in a manner that is relevant,
  clear, comprehensive, and conscious of possible
  bias in the information selected
• of Bias

58
Example 2

• Imagine and seek out a variety of possible goals,
  assumptions, interpretations, or perspectives
  which can give alternative meanings or solutions
  to given situations or problems

59
Example 3

• Formulate and test hypotheses by performing
  laboratory, simulation, or field experiments in
  at least two of the natural science disciplines
  (one of these experimental components should
  develop, in greater depth, students laboratory
  experience in the collection of data, its
  statistical and graphical analysis, and an
  appreciation of its sources of error and
  uncertainty)

60
Core Curriculum

• 6. What an assessment plan looks like

61
Relate
Speak
Listen
Participate
Write
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
62
Core Curriculum

• What an assessment plan looks like
• How assessment works

63
How Assessment Works
Cycle 2
Cycle 3
Cycle 1
O U T C O M E
New / Revised LEvent 1 New / Revised LEvent
2 New / Revised LEvent 3
New / Revised LEvent 1 New / Revised LEvent
2 New / Revised LEvent 3
LEvent 1 LEvent 2 LEvent 3
Process Reflection Compare results against
Benchmarks, Standards, Targets,Past Performance
component component component component BASELINE
component component component component
component component component component
64
Core Curriculum

• What an assessment plan looks like
• How assessment works
• Why you need to define your outcomes

65
teacher5
teacher4
teacher2
teacher1
teacher3
Speaking
volume
eye contact
gestures
sources
transitions
poise
style
rate
examples
verbal variety
conclusion
appearance
evidence
organization
attention getter
66
Can our students deliver an effective Public
Speech?
volume
eye contact
gestures
sources
transitions
poise
style
rate
examples
verbal variety
conclusion
appearance
evidence
organization
attention getter
67
Core Curriculum

• What an assessment plan looks like
• How assessment works
• Why you need to define your outcomes
• How to implement the plan

68
Phase Four
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Course 5
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
69
Core Curriculum

• What an assessment plan looks like
• How assessment works
• Why you need to define your outcomes
• How to implement the plan
• 10. What to do with the data

70
Patterns of Evidence
DP1
DP2
DP3
71
Interpreting Data

• Consistency - over time
• Consensus - different populations
• Distinctiveness - different situations/ variables
  / items

72
Consistency

• Examines the same practice of and individual or
  group over time
• Key question
• Has this person or group acted, felt, or
  performed this way in the past / over time?

73
Consistency
How well are students performing on
the departmental learning outcome measures?
High performance
Low performance
04
05
00
01
02
03
74
Consensus

• Comparison to or among groups of students
• Variation between disciplines, gender, other
  demographic variables
• Key questions
• What is the general feeling, outcome, attitude,
  behavior?
• Do other groups of people act, perform or feel
  this way?

75
Consensus
How well are students performing on
the departmental learning outcome measure?
High performance
Low performance
Females
Males
Transfers
OTA
76
Distinctiveness

• Examines individual or cohort perspectives across
  different outcomes
• Key Question
• Does a person or group perform equally as well on
  different outcomes?

77
Distinctiveness
How well are our students performing on the
learning outcomes?
S P E A K I N G
High Performance
A N A L Y S I S
T H I N K I N G
R E S E A R C H
E THICS
W R I T I N G
Low Performance
78
Fundamental Question 1

• Do we have enough data from which to really draw
  conclusions?

79
Fundamental Question 2

• Does the data represent an identifiable trend in
  the level of activity / achievement /
  accomplishment?

80
Fundamental Question 3

• Does the data represent an acceptable level of
  activity / accomplishment / achievement given our
  mission and values?

81
Fundamental Question 4

• Are the differences in the sub-populations
  acceptable?

82
Fundamental Question 5

• What can we do about it?

83
Really Big Mistakes
84
Big Mistakes in Assessment

• Assuming that it will go away
• Allowing assessment planning to become gaseous
• Assuming you got it right -- or expecting to get
  it right -- the first time
• Not considering implementation issues when
  creating plans

85
Big Mistakes in Assessment

• Borrowing plans and methods without acculturation
• Setting the bar too low
• Assuming that youre done and everythings OK, or
  rushing to Close the Loop
• Doing it for accreditation instead of improvement

86
Big Mistakes in Assessment

• Confusing program effectiveness with student
  learning
• Making assessment the responsibility of one
  individual
• Assuming collecting data is Doing Assessment

87
Assessment 101The Core Curriculum

• Susan Hatfield - Winona State
• UNC Wilmington
• November 2005
• SHatfield_at_winona.edu