ASSESSMENT IN GP/FM

1
ASSESSMENT IN GP/FM

• Mladenka Vrcic-Keglevic, MD, PhD
• Head of Family Medicine Department, Medical
  School, Univerity of Zagreb
• mvrcic_at_snz.hr

2
Content of the session

• Definition of the terms
• How to use assessment (summative, formative)
• Assessment methods (AM)
• What can be assessed by different AM
• Metrics characteristics of AM
• ..
• Interactive session

3
Definition of the terms

• Evaluation A process that attempts to
  systematically determine the relevance,
  effectiveness, and impact of activities in light
  of their objectives. Can be related to structure,
  process or outcome of (programme, person).
• Assessment A system of evaluation of
  professional accomplishments in light of
  educational objectives, using defined criteria
  and usually including an attempt at measurement
  either by grading on a rough scale or by
  assigning numerical values.
• 
  
• 
  AMEE Glosary of Medical Education
  Terms

4
Well defined educational objectives - prerequsite
for assessment

• Example for this session
• At the end of this session participants will be
  able
• To named at list three differences between
  summative and formative assessment
• To make a list of at least three written AM
• To name the most effective AM to assess clinical
  skills
• To describe the most effective AM to assess
  attitudes

5
How to use assessment?

• Summative This is usually undertaken at the end
  of a training programme or teaching course and
  determines whether the educational objectives
  have been successfully achieved. With summative
  assessment the students usually receives a grade
  or a mark. Exam
• Formative This is testing that is part of
  developmental or ongoing teaching / learning
  process. It should emanate from a wish to foster
  learning and understanding. It should include
  delivery of feedback to the student.

6
Summative - Examination

• What the exams are?
• For students A difficult and unpleasent
  steeplechase to run on a way for diploma
• For teachers A less desirable teaching activity
• For public An important protection from
  un-competent doctors

7
Summative assessment

• The reasons
• A statement of achievement - university degree
  (diploma)
• An entrance requirement to an educational
  institution
• A guide as to the wisdom of continuing with
  further study
• A certification of competence public
  responsibility (licence)
• A determinant of programme effectiveness

8
Assessment methods (AM)

• Written
• MCQ - Multiple Choice Questions
• Un-Q An-cued Questions
• Essay
• MEQ Modified Essay Questions
• EMI - extended matching items
• PMP Patient Management Problem

• Oral, viva
• Traditional practical
• OSCE Objective Structured Clinical Examination
• SP - Standardised Patient
• Portfolio
• ACC Assessment of Clinical Competence

9
What can be assessed by different AM

• Factual knowledge
• Interpretations
• Problem-solving skills
• Ethical
• Clinical skills
• Emotional reactions
• Communication

10
What can be assessed?
Fact Inter P-S Ethi Skill Emo Com
Oral /- - - - -
Practic. - - -
MCQ - - - - - - - - - -
Un-Q - - - - - - - - - -
MEQ - - - - - - -
Essay - - - - - - - -
OSCE - - - -
PMP - - - - - - - -
ACC
11
Metric characteristics of AM

• Validity - the degree to which a measurement
  instrument truly measures what it is intended to
  measure
• Reliability it is an expression of the
  precision, consistency and reproducibility.
  Ideally, measurements should be the same when
  repeated by the same student or made by the
  different assessors.
• Relevance it is a degree to which the
  assessment questions and educational objectives
  are in concordance
• Objectivity it is a level of agreement among
  independent assessors (experts) about the right
  answer to certain question

12
Metrics characteristics?
Validity Reliability Relevent Objectivity
Oral - - - - - - - -
Practic. - - -
MCQ
Un-Q
MEQ
Essay - - -
OSCE
PMP/SP
ACC - - -
13
Practicality?

• Preparation resources (time, human)
• Easy to perform, aplay
• NC number of candidates
• NC/teacher number of candidates per one teacher

14
Practicality
Preparation Ease to applay NC- No of candidates NC / teacher
Oral - - - - - -
Practic. - - - -
MCQ
Un-Q
MEQ - -
Essay - -
OSCE
PMP/SP -
ACC - - - -
15
MCQ Multiple Choice Questions

• Difficulty index
• Discrimination index
• Minimum pass level

16
Indeks of difficulty

• QD (G P)/N
• QD questions difficulty
• G number of right answers in good group
• P number of right answers in poor group
• N number of students in both groups

a b c d e 0 U
1
2
3
17
Difficulty Index

• 0,3 0,5 0,6
  0,7
• ------/---------------(------------)----------/--
  -------
• 
  recommended
• ------------------------------------
  --------------------------
• 
  acceptable
• too difficult
  too difficult
  

18
Discrimination Index

• DI 2 (G-P)/N
• DI discrimination Index
• G number of right answer in good group
• P number of right answer in poor group
• N number of students in both group

a b c d e 0 U
1
2
3
19
Diskrimination Index

• 0.15 0.25 0.35
• ----------/----------/----------/---------
  -
• throw off to check good
  excelent

20
What AM to use in reality?

• There is no perfect AM
• A choice depends of what is to be measured
  educational objectives (knowledge, skills,
  competences, attitudes)
• Careful preparation, trained assessors
• Multiple methods better then one

21
Specialistic exam in GP in Croatia

• Should be submeted in advance
• Portfolio
• 8 essays
• Questionary of professional competence signed
  by trainer
• Trainers report about trainees achievements
• Trainees report How to organise my practice,
  what to invent?
• Medical records (videos) of 3 patients / family

22
Specialistic exam in GP in Croatia

• MCQ test 150 questions
• OSCE 20-25 stations
• Oral jury (3 members) 1 academic GP 1 well
  recognised GP (publication) 1 trainer

23
Formative assessment

• The reasons
• Information for the student about his/her
  achievement of educational objectives
• Repetitive progress measurement
• Discover week pointsteachers support
• Help to a teacher to correct programme
• The results should not be used in summative
  assessment

24
Formative assesssment -which AM?

• The same as for summative progress test
• Protocols for observing clinical work
  (consultation, communication, p-s abilities,
  professional competences)
• Checklists
• Diaries what has been done
• Appraising projects, audit
• Portfolio
• Self and peer assessment

25
Formative assesssment

• Feedback
• Feedback
• Feedback
• Feedback
• Feedback

26
Educational objectives in this session achieved?

• At the end of this session participants will be
  able to
• name at least three differences between summative
  and formative assessment
• make a list of at least three written AM
• name the most effective AM to assess clinical
  skills
• describe the most effective AM to assess
  attitudes