Assessing When Numbers Don

1
Assessing When Numbers Dont Count

• Binghamton University
• March 23, 2007

2
Todays Objectives

• Define what a discourse-based, or qualitative
  assessment method is
• Review a few of these methods
• Work with some case studies to better understand
  how these methods might be used
• Discuss the uses and limitations of these
  methods, especially with regard to assessment

3
The Pressure to Assess

• This weekSpellings Commission meeting to discuss
  several issues. One of them is student learning
  outcomes
• There is consistent pressure to use standardized
  tests and surveys we are currently using one of
  them, the NSSE
• Through the use of standardized tests and
  surveys, we gain perspective on how we compare to
  other institutions
• We might be able to gain a value added
  perspective

4
Weaknesses of the Standardized Test/Survey
Approach

• We obtain a macro perspective, but may not gain
  a micro perspective
• We have little control over the questions
• Value added is still difficult to establish
  with standardized tests
• Faculty see little value in using standardized
  tests, especially in interdisciplinary studies
  and the liberal arts
• Standardized tests/surveys often assume they can
  measure most of what concerns faculty in respect
  to teaching and learning

5
Forging the Middle GroundDiscourse-Based
Assessment Methods

• Allow for the discovery of the unanticipated
• Relevant to interdisciplinary study
• Maximizes faculty/staff input when discourse is
  highly valued
• Good to use when the number of objectives
  outweigh the amount of time available to assess
  student learning
• Great contextualizer
• When performed carefully and compared with other
  methods, great return on investment

6
Types of Qualitative Assessment

• Focus groups
• Expert panels
• Open-ended surveys
• Ethnographic studies (participant observations)
• Portfolio reviews
• Primary trait scoring
• Delphi panels

7
Qualitative Assessment is an Inductive Process
Defined Focus
Observations
Analysis
Summary Report
Comparison with Other Assessments
Action
8
Expert Panel

• Type of focus group
• Focus is on (for assessment purposes) a
  particular assignment or performance
• Not a simple conversation is done methodically,
  with precision, and is systematic
• Often involves convenience or random samples of a
  homogeneous population
• Must include carefully written questions
• Might include a survey or other assessment
  technique as part of the process

9
Kruegers 10 Quality Factors in Focus Group
(Expert Panel) Research

• Clarity of purpose
• Appropriate environment
• Sufficient resources
• Appropriate participants
• Skillful moderator
• Effective questions
• Careful data handling
• Systematic and verifiable analysis
• Appropriate presentation
• Honoring the participant, clarity, and method

10
Expert Panel Procedure
Write Questions
Use Additional Method?
Select Focus
Select Experts
Conduct Expert Panel How to Assess Agreement?
Logistics
Report Results
11
Final Thoughts on Expert Panels

• Good method of assessing ineffable outcomes
• Better when expert panel questions and
  conversations are grounded in standards and/or
  student learning objectives
• Also good when specific focus is on a particular
  assignment or performance
• It is advisable to use a secondary method either
  prior to or during expert panel
• Does not control for anonymity among respondents

12
Primary Trait Scoring

• Focus is upon one particular assignment,
  performance, etc., that is reflective of several
  aggregate student learning outcomes
• Rate each outcome according to a scaleex.,
  proficient, satisfactory, unsatisfactory
• Idea is to look at trends, not numbers, that
  spark discussion

13
Primary Trait Scoring--Procedure

• Choose an assignment in which students
  demonstrate summative knowledge, skills, or
  competencies
• Carefully rate student performance according to
  the scale
• Place checkmarks in each column
• Look for visual trends
• Discuss why these trends occur, what basis these
  rating occurred, and what specific issues are
  revealed through the analysis
• Combine with other findings, or make plans for
  action

14
Example of Primary Trait Analysis
Element Excellent Satisfactory Unsatisfactory
Demonstrate an ability to research the financial integrity of a business plan through accounting/financial analysis vvvv vvvvvv vv
Demonstrate an ability to assess a plans organizational integrity that is, a business plan demonstrates how an organization will be built and sustained to under gird the success of the business being organized vv vvvv vvvvvv
15
Primary Trait AnalysisFinal Thoughts

• Method to get faculty or staff to talk about what
  assessment results mean
• A good starting point toward developing a rubric
• Enables discussion, which can lead to further
  discovery
• Simple, stress free, and easy

16
Delphi Panel Introductory Exercise

• Divide into 3 groups
• Get out piece of paper and individually write
  down, what do students have the most difficulty
  with when they first come to college (as
  first-year students)?
• Try to create frequency countscombine like
  answers and tally them
• Discuss

17
Questions for Groups

• What do these say about the difficulty students
  might have when they start?
• Take a look at the most popular answerdo these
  ordinarily achieve majority vote status?
• Even in cases where majority vote status is
  achieved, might less popular answers indicate
  group consensus?

18
Introduction to the Delphi Method

• Combination of at least 3 methodsopen-ended
  survey, closed-ended survey, and expert panel
• Unlike expert panel, attempts to maximize
  anonymity of respondents to control for power
  dynamics among these respondents
• Assumes highly motivated groups of experts
  (faculty or staff) willing to participate in more
  than one round of questions

19
Introduction to the Method

1. Find homogeneous group of experts who can comment
   either on one assignment or specific student
   learning outcomes
2. Create an open-ended survey in which respondents
   are asked to identify strengths and weaknesses in
   student performance in reference to specific
   standard or student learning outcome
3. Content analyze responses by combining like
   responses, placing how many times each was
   mentioned in parentheses
4. Cut and paste these onto a survey, and ask
   respondents to indicate to what extent they agree
   with each on a 4 or 5 point scale
5. Report those responses that indicate consensus
6. If needed, move to 3rd round, in which
   respondents rank these consensus items

20
Strengths of Delphi Method

• A way of addressing ineffable outcomes
• Can be used to designate the most agreed-upon
  student learning objectives that faculty have
  communicated
• Can be used to gather information from employers,
  internship supervisors, alumni, etc. about
  specific items of interest

21
Limitations of Delphi Method

• Can be time consuming
• Takes some knowledge of statistics
• Not a method that can be used by itself usually
  results need to be compared with direct
  assessments of student learning

22
Todays Activities

• Separate into three groups select group note
  taker
• If you have not already, read the case study
  packets
• As a group, discuss questions at end of case
  studydebate, applaud, etc.do something active
• Write answers on provided sheet of paper