Criterion Based vs' NormBased Evaluation - PowerPoint PPT Presentation

Title: Criterion Based vs' NormBased Evaluation


1
Criterion Based vs. Norm-Based Evaluation

• David L Gaspar MD
• October 18, 2008

2
Evaluation of Students

• Goals- This session is designed to enable
  participants to
• 1. Understand the definition and purpose of
  evaluation
• 2. Develop a framework to make evaluation
  efficient and useful
• 3. Understand and apply criterion-based
  evaluation the R.I.M.E. method

3
Evaluation Basics

• Definition
• Evaluation is the process of making judgments
  based on factual information and observations in
  order to rate, rank, or assess an individuals
  status at a given point.
• Purposes of Evaluation
• 1. Summarize performance at a given point in
  time.
• 2. Provide information for planning future
  educational experiences.
• 3. Communicate summary information to other
  parties.

4
Framework for Planning Clerkship Evaluation

• Before Clerkship
• 1. Understand medical school expectations and
  forms.
• 2. Review clerkship goals and objectives.
• During Clerkship
• 1. Gather information from multiple sources
  provide feedback.
• 3. Use systematic method of recording.
• End of Clerkship
• 1. Prepare for final evaluation.
• 2. Schedule and conduct summary meeting.
• 3. Complete and submit final evaluation.

5
Tips to Facilitate Evaluation

• Dictate or write a student progress note. (end of
  day, when procedures occur-behaviours)
• Use student log books.
• Review students written record. (copy progress
  notes at the beginning, middle, and end of
  clerkship)
• Use student self-assessment forms.
• Use computer printout of students patient
  profile.

6
Evaluate

• End of Clerkship
• 1. Review evaluation criteria and form.
• 2. Review student performance data.
• Schedule and conduct final evaluation session
  with student.
• Have some backbone
• 5. Complete and return the evaluation form
  promptly. (Keep a Copy!!!)

7
Norm-Based vs. Criterion-Based

• Any ideas????

8
Norm-Based vs. Criterion-Based

• NORM-BASED EVALUATION
• Norm-based assessment compares individuals
  performances to one another, resulting in a
  ranking of individuals.
• Medical Knowledge

9
Norm-Based vs. Criterion-Based

• NORM-BASED EVALUATION
• This traditional form of assessment has been
  widely used to select individuals for further
  education or for jobs and is what most students,
  residents, fellows and faculty have been exposed
  to throughout their training

10
Norm-Based vs. Criterion-Based

• NORM-BASED EVALUATION
• Since norm-based assessment provides no
  information about the observed quality of the
  performance, it isnt useful for providing
  feedback or improving performance.
• The raters task ends up distinguishing one
  individuals performance from anothers.

11
Norm-Based vs. Criterion-Based

• NORM-BASED EVALUATION
• Raters will vary in their assessments depending
  on their experience with using a norm-based
  scale, and their experience with the group being
  evaluated

12

• NORM-BASED EVALUATION

Dr. Thomas, who has 15 years of experience
working with and evaluating students, is due to
evaluate John, his student over the last 2 weeks.
He has observed John performing a brief history
and physical examination, has heard daily oral
presentations, and has interacted with him on
multiple occasions to assess his fund of
knowledge. Dr. Thomas has a clear expectation
based on this experience for what an honors
student is. He believes that John is a very good
student with great enthusiasm, but not quite
functioning at an honors level and will therefore
complete his evaluation in a manner that reflects
what he thinks will result in a high pass
grade- on a 6-point scale. mostly 5s and some
6s. 1 poor performance and 6excellent
performance,
13

• NORM-BASED EVALUATION

Dr. Thomas evaluation 1 poor performance and
6excellent performance,
14

• NORM-BASED EVALUATION

The next attending to work with John, Dr. Smith,
has just completed residency. She too has been
working with John for 2 weeks and has observed
him doing a brief history and physical
examination, has heard daily oral presentations,
and has interacted with him on multiple occasions
to assess his fund of knowledge. She is an
enthusiastic teacher and believes that students
should get honors if they are active participants
in rounds, enthusiastic about learning and
generally helpful. She went to a medical school
where most students got honors if they did these
things and she thinks this is a good policy. She
has heard from the medical students and residents
that she has worked with that to get honors, she
must give John all 6s on a 6 -point scale 1
poor performance and 6excellent performance,
15

• NORM-BASED EVALUATION

Dr. Thomas evaluation 1 poor performance and
6excellent performance,
16

• NORM-BASED EVALUATION

What grade does John deserve? More
importantly, what is Johns level of competence?
17

• NORM-BASED EVALUATION

Answer Hard to tell..This example provides
insight into some of the challenges of norm
-based assessment. Now lets learn about an
alternative means of assessment.
18
How do we evaluate the learner with a
criterion-based system?

• There have been a variety of methods used over
  the past several decades
• In 1999 L. Pangaro developed RIME a new tool
  for evaluating the level of your learner more
  specifically for medical students

19
RIME

• Reporter
• Interpreter
• Manager
• Educator
• Before RIME Observer/Pre-Reporter
  (pre-clinical)

20
RIME

• Reporter - Consistently good in interpersonal
  skills, reliably obtains and communicates
  clinical findings. Written and spoken
  presentations are clear and organized.
• Interpreter Able to prioritize and analyze
  patient problems, develop a differential
  diagnosis and next steps in workup. Data is not
  just data (reporter), but has a clinical
  significance.

21
RIME

• Manager Consistently proposes reasonable
  diagnostic and therapeutic options, incorporating
  patient preferences. They can take their
  knowledge, integrate it with the findings for a
  given patient and form a plan that prioritizes
  within and amongst the problem list.
• Educator Consistent level of knowledge of
  current medical evidence demonstrates
  self-directed learning (practice based
  improvement and learning) and contributes to
  others education.

22
RIME Changes with Time

• You can use RIME to track the progress/growth of
  a student over the course of their clinical years
  or within a given year.
• Early MS3 Should report well and begin to
  interpret.
• Late MS3 Should be a good interpreter
• Sub-I should have the above and begin to manage
  patients appropriately.
• Interns should become astute managers
• R2s and beyond progressive depth of management
  and becoming educators

23
RIME Criterion-based evaluation

• Criterion -based assessment begins with an
  established set of criteria for successful
  performance and compares an individuals
  performance to these criteria.

24
RIME Criterion-based evaluation

• Rating an individuals performance against the
  criteria provides feedback on the quality of
  performance. Multiple individuals may
  successfully meet the criteria as one
  individuals performance is not constrained by
  anothers performance. Agreement between raters
  is high as the standard for comparison is stable
  and based on observable behaviors. In other
  words, the rater is responsible for documenting
  what the specific skills of the individual are
  based on their observed performance. They are NOT
  responsible for comparing them to anyone else, or
  for trying to determine a grade.

25
RIME Criterion-based evaluation
In the scenario just described, both Dr. Smith
and Dr. Thomas have observed Johns oral
presentations. When they are asked to complete
the evaluation of John they are asked to rate his
abilities based on observable behaviors related
to his oral presentations. Johns oral
presentations are accurate and objective. But,
they sometimes contain extraneous information and
important information gets lost in the
presentation. Please complete the evaluation
below as if you were Dr. Thomas, with 15 years of
experience teaching medical student
26
RIME Criterion-based evaluation
Would you evaluate the student any differently if
you were Dr. Smith?
27
RIME Criterion-based evaluation
John is able to derive a rudimentary differential
diagnosis and assessment for patients presenting
with shortness of breath and chest pain with only
minor assistance from an intern or resident. But,
he has no idea how to approach the presenting
problems of acute renal failure and hyponatremia.
For each of these latter clinical situations, the
intern needed to completely develop the
differential diagnosis and teach it to John, who
was then able to repeat it at attending rounds.
He is only able to develop management plans with
significant help from the intern. Each of these 4
conditions is common on your rotation and
required clinical entities for the
clerkship. How would you rate him on the
following scale?
28
RIME Criterion-based evaluation
John is able to derive a rudimentary differential
diagnosis and assessment for patients presenting
with shortness of breath and chest pain with only
minor assistance from an intern or resident. But,
he has no idea how to approach the presenting
problems of acute renal failure and hyponatremia.
For each of these latter clinical situations, the
intern needed to completely develop the
differential diagnosis and teach it to John, who
was then able to repeat it at attending rounds.
He is only able to develop management plans with
significant help from the intern. Each of these 4
conditions is common on your rotation and
required clinical entities for the
clerkship. How would you rate him on the
following scale?
29
And Then Its in My Court