Written Submission for Summative Assessment

1
Written Submission for Summative Assessment

• Audit
• Dr. Jane Mamelok
• Associate Director GP education

2
Completed Audit Cycle

• Run in West of Scotland since 1997
• Competencies tested
• The ability to review and critically analyse the
  practitioners own working practices
• and manage necessary changes appropriately
• Registrar has to initiate the audit
• Encourages a closer reflection of usual general
  practice

3
Audit Marking Criteria

• 8 point marking scale completed audit cycle
• Previously 5 point marking scale, Oxford stage V
  audits (planning stage only, not assessing the
  change)
• All new entrants - starting after February 2001
  must submit for marking under the 8 point scale.

4
Audits Marked Against 8 Criteria

• Reason for choice of audit
• Criterion/criteria chosen
• Standards set
• Preparation and planning
• Data collection 1
• Changes to be evaluated
• Data collection 2
• Conclusions

5
Marking Audit

• 8 criteria
• All criteria must be present to pass - marked
  present or absent
• 2 first line markers
• Either can refer
• 2 second line markers, (pass or return for
  resubmission. Resubmission is a pass or refer)
• National panel

6
Feedback

• Projects are currently marked by Yorkshire
  Deanery - feedback on failed criteria is given by
  them.
• NW Deanery audits referred - copies to Associate
  Director who gives formative feedback to trainer
  on general principles of teaching GPRs audit and
  preparing them for summative assessment.

7
(No Transcript)
8
Why audits fail?

• 2002/2003 - 115 GPRs
• 13 audits referred for resubmission
• 1 resubmission failed at deanery level but passed
  by national markers
• 11 referral rate

9
(No Transcript)
10
(No Transcript)
11
Why audits fail?

• Commonest criteria
• Wooly criteria, difficult to define and measure.
• Cascade effect, wooly criteria suggests poor
  planning and lack of understanding of audit
  process
• Wooly criteria often lead to difficulties in
  collecting relevant data.

12
Reason for the Choice of Audit

• Not a common criterion for failure
• Needs to be relevant to general practice
• Selected and owned by the registrar
• Subject appropriate amenable to audit
• Some GPRs try to adapt, research/project work
• Difficulty in defining criteria standards

13
Criteria

• Common cause for failure
• Clear focused audit question is the key
• Keep criteria simple
• succinct clear - avoid woolyness
• Criteria defined should be supported by current
  evidence
• reference to lit. search should be made

14
Wooly criteria

• Audit title
• Have hysterectomised women 40 - 65 had the
  opportunity to discuss osteoporosis risk ?
• Criterion - evidence of consideration or
  discussion of risk of osteoporosis
• Standard 80 patients in defined population.

15
Remedy

• Keep it simple
• clearly defined audit question
• clear, defined criteria.

16
Criteria Standards

• Do Type 2 diabetics meet UKPDS recommendations
  for cholesterol levels.
• Criteria
• Type 2 diabetics should have a cholesterol
  recorded in the last 12/12 at lt 5mmol/l - ref
  UKPDS
• Standard
• 70 - based on UKPDS seen in practice

17
Standards

• Need to be realistic and achievable
• Evidence based or
• Agreed after discussion with the practice team
• Useful to clearly lay out the standards for each
  data collection and the standards by which they
  are assessed

18
Standards

• If it is difficult to define a standard look at
  best practice or do a mini survey of current
  trends in your own practice.
• E.g. Consider an audit on epilepsy care looking
  at what proportion of patients with epilepsy have
  a correct diagnosis recorded on CD register.
• Local clinical governance stats might tell you
  that most practices achieve only say 60
  patients correct on register.
• A mini survey might show you that only 40
  patients are on register, therefore setting a
  standard 50 - 60 would be reasonable.

19
Preparation Planning

• Another common fail area
• Not just a description of methods
• Some account of the decision making processes in
  planning the audit including suitable criteria
  defined
• Must include teamwork aspect
• e.g. if an audit looks at diabetic care, the
  nurse running the clinic should be involved in
  the planning as there may be implications for her
  of any changes
• Discussion of appropriate use of resources
• planning delegation negotiation, implementation.

20
Data Collection

• Common fail criterion
• Data collection must relate to criteria assessed
  - easier if criteria clear audit question
  focussed
• Give absolute figures not just percentages and
  make sure they tally!!
• State total sample size
• Clear comparison to the standard set

21
Data collection

• Charts are good but the supporting data table
  should be included.
• Data collection 1 and 2 should compare data sets
  assessed by the same criteria
• Within a reasonable stated time scale
• Compare like with like
• A clear statement of the 2 data sets is helpful

22
Data sets example

• Data collection 1
• Total sample Type 2 diabetics in GP care with a
  cholesterol measurement in last 12 months
• Data collection 2
• Type 2 diabetics identified in 1 with no
  cholesterol done or gt5, plus any new diabetics
  both assessed after implementation of the change

23
What about sample data sets

• If you use a sample data set e.g. 50 patients
  with established CHD for data collection 1.
• Data collection 2 must be a different sample
  (different 50 patients)from the same total IHD
  patients, including new cases
• You cannot just measure the effects of change on
  the 1st data set not meeting your audit criteria
  e.g. only reaudit 25/50 original patients with
  cholesterol gt 5, you must sample a different 50
  patients from the same total population.

24
Implementing the change

• Discuss the data commenting on whether standards
  met and identifying areas for change
• Detailed plan of the changes needed
• with actual examples
• Plan of how the change is to be implemented and
  monitored
• Teamwork element important
• Discussion of the impact of the change - have
  standards improved?

25
Conclusions

• Summary of the audit, it methods and findings
• Summary of the changes and impact
• Summary of the lessons learned
• Implications for GPR
• Implications for practice team
• Keep it succinct

26
Summary

• Clear simple, focused audit question
• Clear criteria - evidence based
• Realistic clearly stated standards that can be
  justified
• Evidence of teamwork in the preparation
  planning
• Sound data relating to the criteria, clear time
  scale between the 2 data collections comparing
  like with like
• Describe the changes and their implications
• Conclusions - clear summary with lessons learned.