Implementing Work-Based Assessments

1
Implementing Work-Based Assessments

• Professor T.Masud
• Nottingham University Hospitals NHS Trust

2
Overview- Curricula and Assessments for Training
CCT in Geriatric Medicine
JRCPTB Cert. in GIM L2
Selection
Allocation
ST1 ST2 Core Training GIM (Acute) L1
Curriculum
F1 F2 FP Curriculum
ST3 ST4 ST5 ST6 ST7 Speciality
Curriculum
GIM (Acute) L2
Curriculum
Generic C. L1
Generic Curriculum L2
MRCP Parts 1, 2 (written), PACES
KBA (specialist exam)
WORK BASED ASSESSMENTS
Tooke report recommends separating F1 and F2
joining F2 to ST12
3
Generic Curriculum - builds on Foundation
Curriculum(competencies categorised into
Knowledge, Skills, Attitudes Behaviour)

• Level 1 (Mandatory CT Competencies)
• 1.1 History taking, examination, record keeping
• 1.2 Time management and Decision Making skills
• 1.3 Good Quality Care and Patient Safety
• 1.4 Infection Control
• 1.6 Valid Consent
• 3.1 Communication wih patients within a
  consultation
• Focus Area 4 Working with Colleagues
• Focus Area 6 Professional Behaviour

4
Generic Curriculum - builds on Foundation
Curriculum(competencies categorised into
Knowledge, Skills, Attitudes Behaviour)

• Level 2 Competency Areas
• 1.5 Health Promotion and Public Health
• 1.6i Medical Ethics and Public Health
• 1.6iii Legal framework for Practice
• 1.7 Ethical research
• 1.8 Managing Long Term Conditions and
  Promoting Patient Self-Care
• Focus Area 2 Governance Maintaining Good
  Clinical Practice
• 3.2 Breaking Bad News
• 3.3 Complaints and Medical Error
• Focus Area 5 Teaching and Training

5
Curriculum for GIM (Acute Medicine) Levels12
(Categorised into levels 1 2 by
Knowledge, Skills , AttitudesBehaviour)

• Emergency Presentations
• Cardio-respiratory arrest
• Shocked patient
• Unconscious patient
• Anaphylaxis

Top 20 Common Medical Presentations
Abdominal Pain Acute back pain
Blackout/Collapse Breathlessness Chest pain
Confusion- acute Cough
Diarrhoea Falls
Fever-
Fits/seizures Haematemesis/Melaena Headache
Jaundice
Limb pain/swelling Palpitations
Poisoning
Rash Vomiting
/nausea Weakness/Paralysis
6
Curriculum for GIM (Acute Medicine) Levels12
(Categorised into levels 1 2 by
Knowledge, Skills , AttitudesBehaviour)
Other important presentations
SYSTEM SPECIFIC COMPETENCIES
Abdo mass/hepatosplenomegaly Abdo
swelling/constipation Abnormal sensation
Aggressive/disturbed behaviour Alcohol/substance
dependence Anxiety / panic behaviour Bruising
Chance
findings Dialysis
Dyspepsia
Dysuria
Genital
discharge ulceration Haematuria
Haemoptysis
Head Injury
Hoarseness and stridor Hypothermia
Immobility
Involuntary movements
Joint swelling
Lymphadenopathy
Loin pain
Medical complics of acute illness/surgery
Medical problems
in pregnancy Memory loss
Micturition (difficult)
Neck pain
Non-organic physical symptoms
Polydipsia
Polyuria
Pruritis
Rectal bleeding
Skin and mouth ulcers
Speech disturbances
Suicidal ideation
Swallowing difficulties
Syncope and presyncope Unsteadiness
/balance problems Visual disturbance
Weight loss

Allergy
Cancer palliative care
Cardiovascular medicine Clinical pharmacology
Dermatology Diabetes
and endocrinology Gastroenterology / hepatology
Haematology
Immunology
Infectious diseases
Medicine in the Elderly Musculoskeletal system
Neurology Psychiatry
Public Health /
H.Promotion Renal medicine
Respiratory medicine
Investigation Competencies
Procedural Competencies
7
Geriatric Medicine ST Curriculum (Jan 2007) 1

• Primary learning objectives (achieve competencies
  in) various
  settings
• Perform Comprehensive Geriatric Assessment
• Diagnose and manage acute illness in old age
• Diagnose and manage those with chronic disease
  and disability
• Provide rehabilitation with the multidisciplinary
  team to an older patient
• Plan the transfer of care of frail older patients
  from hospital
• Assess a patients suitability for and provide
  appropriate care to those in long term
  (continuing) care in the NHS or community
• Apply knowledge and skills of a competent
  geriatrician in an intermediate care and /or
  community setting

8
Geriatric Medicine ST Curriculum (Jan 2007) 2

• 8. Assess and manage older patients
  presenting with the common geriatric problems
  (syndromes)
• a. Falls- with and wihout fracture
• b. Delirium
• c. Incontinence
• d. Poor mobility
• 9. Demonstrate competence in following
  subspecialities
• a. Palliative care
• b. Orthogeriatrics
• c. Old Age Psychiatry
• d. Specialist Stroke Care
• 10. Be competent in
• a. Research methodology (basic)
• b. Ethical principles of research
• c. Critical appraisal of medical
  literature
• d. Preferably to have personal
  experience of research
• basic
  science or clinical (health service)

9
Assessment Plan in Geriatric Medicine - Overview

• MRCP PACES
• KBA (SE)
• Work Based Assessments
• Mini-CEX, DOPS, ACAT, CbD, MSF, PS
• Assessment of Generic Areas
  
• Research (portfolio, supervisor reports,
  publications)
• Audit (portfolio, audit reports, supervisors
  reports)
• Clinical Governance (portfolio, supervisor
  reports)
• Teaching (portfolio, assessed teaching)
• Regular Appraisal by Educational Supervisor
• (including Educational Supervisor Reports)
• Training Record
• educational supervisor consulatant trainer
  reports
• completed signed work based assessments

10
Work Based Assessment Methods

• Mini-Clinical Evaluation Exercise (mini-CEX)
• Direct Observation of Procedural Skills (DOPS)
• Multi-Source Feedback (MSF)
• Case-Based Discussions (CbD)
• Patient Survey (PS)
• Acute Care Assessment Tool (ACAT)

11
Mini-CEX (ST3-ST7) n25
4 Acute (1 ward round) 2 Rehab (1 ward round) 1
Pre-op Orthogeriatric 1 Post-op Orthogeriatric 2
Chronic disease (clinic eg DM, OA) 1 MDT chair
(discharge) 1 Continence 1 Falls 1 Movement
disorder
1 Delirium / Depression 1 Old Age Psychiatry (HV
/ Ward referral) 1 Osteoporosis / metabolic bone
disease 1 Comprehensive Geriatric Assessment 1
Intermediate care / Home visit 1 Continuing
care 1 Day Hospital 3 Stroke (acute WR, Rehab WR,
TIA clinic 1 Palliative care (break bad news)
12
CbD (ST3-ST7) n26
4 Acute (diagnosis, Mx, prescribing) 2 Rehab 2
Continuing care ( 1 non-NHS) 1 Evidence Based
Medicine 1 Ethics / Law 1 Health Promotion 1
Complaint 1 Intermediate Care 1 Transfer of Care
problem 1 Delirium
1 Old Age
Psychiatry
1 Depression / dementia 1 Falls

1 Continence 1 Orthogeriatric acute
1
Orthogeriatric - rehab 1 Acute Stroke 1 Rehab
Stroke 1 Neurovascular investigation (TIA) 1
Palliative care 1 Tissue Viability / Hypothermia
13
Mid-Trent Experience of MSF
(360 degree appraisals for SPRS)
14
Overview of Earlier Pilot

• Early pilot June-August 2004
• 18 SPRs in 6 NHS Trusts
• Then 13 SPRs (1 abroad, 1 maternity leave)
• 11 SPRs performed 360 degree appraisals
• Minimal guidance
• Experience discussed at RITAs Sept 2004
  (subjective)
• Questionnaire sent to SPRs in Jan 2005
  (objective)

15
Who decided which people the form should be sent
to?
No.
Ward Manager
16
Did you find the exercise useful?
Yes- extremely useful 1
Yes- quite useful 4
Not sure 3
No- not that useful 2
No- waste of time 1
17
Comments Those who found process useful

• Gave useful feedback on how I am performing on
  the ward
• I think it is a great way of assessing
  clinicians
• Getting positive feedback improved my
  confidence.
• The process informed the appraisal meeting with
  my
  
• educational supervisor

18
Comments Those who found process not useful

• Other people should distribute the forms so the
  SPR does not know who has been asked to complete
  them (x2)
• Meetings with the educational supervisor more
  important than the 360 degree appraisal

19
Further analysis

• Of the 5 SPRs who found the process useful, for
  3 of them others decided who to send the forms
  to.
• Of the 6 SPRS who who did not find the process
  useful or were not sure, all 6 SPRs chose the
  people to send the forms to

20
Conclusion of the Early pilot
Main hypothesis generated The process is more
useful if educational supervisor / consultant
trainer decides to whom the questionnaires are
sent to.
21
Randomisation- by Trust
Randomised to
3 Trusts
3 Trusts
SPRs decides (n8)
ES/Con decides (n7)
Questionnaire sent to SPRs and ESs after the
RITAs in Oct 2005
22

• Mean no of Qs sent 17.8 (range 10-20)
• Mean returns 15.1 (range 10-20)
• Mean response rate 85 (range 55-100)
• No difference between the 2 groups

23
Did you find the process useful?
Total SPR Decides ES/Con Decides
Yes- extremely 2 0 2
Yes- quite 9 4 5
Not sure 2 2 0
No- not that useful 2 2 0
No- waste of time 0 0 0
50 useful 100 useful
Chi sq 4.77, df 1, p0.029 Fishers p0.051
24
Mini-CEX

• 12 SPRs had performed at least 1 Mini-CEX
• 9 found them useful, 3 not
• Egs - Ward round (x4) MDM (x2) Tilt (x1)
  - Rest not stated
  

25
DOPS Patient Satisfaction Qs

• 4 SPRs had performed at least 1 DOPS
• 3 found them useful (Temp pacing, OGD, not
  stated)
• 1 not useful (LP)
• 1 SPR performed a patient satisfaction Q- useful

26
Feedback from Educational supervisors
(5 returns from 7)

• 2 found the 360 degree process extremely useful
• 2 found it quite useful
• 1 not sure

27
Feedback from Educational supervisors
Was the process time-consuming? No-
4 Yesslightly 1 Yes very 0
28
Assessment Plan For Speciality Training in
Geriatric Medicine
Summative
Formative
MRCP KBE(SE) Mini-CEX CbD MSF ACAT PS ESCTReports Audit / Research Teach Asses Cert
ST3 6 Acute 6 Acute 1 ALS
ST4 6 6 1 Acute Audit ALS Res. Meth.
ST5 6 6 1 Audit (eg rehab) ALS Teach
ST6 6 6 PYA Audit (eg IC) Res. present. ALS
ST7 1 2 1 Publication ALS Manag.
TOT 25 26 2 4 2
29
Role of the ARCP
30
Challenges

• Overlap between SpRs and StRs
• StRs proactive
• Educational supervisors proactive
• Consultant Trainers proactive
• New documentation structured
• Time in Job plans
• Useful process or tick-box exercise?

31
Some Suggestions

• StRs and CTs to have forms for CbDs, Mini-CEX
• handy
• 2 CbDs and 2 Mini-CEX per 4 month attachment
• 3 CbDs and 3 Mini-CEX per 6 month attachment
• Importance of Meeting with Education Supervisor
  
• about 1 month before ARCP
• Regular half day / full day teaching sessions
• mapped to curricula

32
Partnership
StR
Educational Supervisor
TPDs
Regional Advisor
Consultant Trainer
Specialist Training Committee
BGS ETC SAC JRCPTB PMETB
NHS Trust
Deanery
33
Assessments! Assessments! More Work Based
Assessments