Missouri Quality Award

1
Missouri Quality Award

• Examiner Preparation Course
• 2006

2
Welcome!

• Introduction
• Name
• Affiliation
• Experience

3
Examiner Preparation Course

• Objectives
• Ethics and Code of Conduct
• Discuss Criteria Changes
• ? Enhance comment writing skills
• ? Calibrate scoring
• ? Stage 1 performance and key milestones
• ? Stage 2 process and key milestones
• ? Judges Stage 2 process
• ? Introduce Stage 3 key milestones

4
Foundation of MQA

• Integrity
• Competence
• Adherence to Code of Conduct
• Ethical Standards

5
Ethical Standards and Conduct (please see page 10
in handbook)

• Avoid Conflicts of Interest
• Maintain confidentiality
• Uphold Code of Conduct

THREE ABSOLUTE ESSENTIALS TO THE CREDIBILITY AND
PRESTIGE OF THE PROGRAM
6
Ethics Exercise
7
Team Assignments
8
Significant Criteria Changes for 2006(please see
page 9 in the criteria book)

• Notes
• Not for profit language
• Item title changes

9
Break
10
Examination Process
You are here
You are here
Receive Applications
Stage1 - Independent Review Individual Scoring by
Examiners
Stage 2 - Consensus Review Examiner Teams Meet
to Develop Scores
11
Stage 1

• Objectives of Stage 1
• Examiners independently produce GREAT
• scorebooks
• Scorebooks have 6 - 10
• comments per Item
• All Items have a score aligned
• with the Scoring Guidelines
• Comments are well written
• Key comments are integrated
• into Key Themes

12
Key Business Factors (please see page 39 in
handbook)

• Significant attributes about the organization
  that influence the way the organization operates.
  
• Provide focus for the assessment.
• Ensure feedback is accurate and meaningful

13
Key Business Factors Examples

• Mission, Vision and Values
• Employee and Staff profile
• Customer and market segments
• Customer requirements
• Competitive Position
• Critical Success Factors
• Strategic Challenges
• Governance Structure

14
Key Business Factors Exercise
15
Team Assignment
16
Producing a Well-Written Scorebook (please see
page 17 in handbook)

• Each Comment
• Addresses only one topic
• Limited to 1 - 3 sentences
• In the form of a finding that follow Comment
  Writing Guidelines
• Is central to the Criteria and KBFs
• Collectively per Item establish where the
  organization is now and the next steps to get to
  the next level of performance

17
Comment Alignment
Determine if process is
systematic
Yes
Effective
Yes
Aligned
Yes
Integrated
Strength
18
Producing a Well-Written Scorebook (please see
page 26 in handbook)

• Scoring
• Strengths should reflect words from Scoring
  Guidelines of where their processes and results
  are now
• OFIs should reflect words from Scoring Guidelines
  of actions to take to achieve performance in the
  next higher box

19
Producing a Well-Written Scorebook (please see
page 18 in handbook)

• and - -
• Your tool to identify candidate
• Best practices and/or Key Theme elements
• Key performance limitations that inhibit reaching
  the next scoring box and/or Key Theme elements
• Are an aid to your teammates in Stage 2 in
  identifying key drivers in all of the Items

20
Producing a Well-Written Scorebook

• After drafting all comments and
• scoring for all the Items, VERIFY
• That the comments make sense
• That the strengths do portray where
• the organization is now in that Item
• That the scoring reflects where the
• organization is now in that Item

21
Observed Pitfalls

• Multiple strengths or OFIs address the same topic
• Being tooooooooo terse (incomplete sentences)
• Being tooooooooo verbose
• Comments at a low level
• Cannot link comments to Scoring Guidelines
• Being prescriptive
• So what being negative

22
Scoring Small Organizations

• Expectations
• Scope of Operation
• Corporate Citizenship
• Information Management
• Comparative Data
• Informal Communication
• Supplier Leverage
• Customer and Market Knowledge
• Documentation

23
Scoring Mentality (please see page 32 in
handbook)

• Stage 1 - Give benefit of the doubt
• Stage 2 - Give benefit of the doubt
• Stage 3 - Show me

24
Item Exercise

• Purposes
• To reinforce comment guidelines for the Process
  Categories
• To identify opportunities for Examiners to
  improve their comment writing skills
• To calibrate Examine Scoring

25
Exercise Requirements

• Exchange Case Study Designated Item with other
  teammates at your table (2 trades)
• Read and annotate your teammates comments
• Note well-written comments
• Provide suggestions of how to improve other
  comments
• Return annotated comments and discuss
• Compare scoring of all teammates at your table
• Discuss the scoring. Identify drivers of the
  score
• Report out on lessons learned

26
Exercise Activities
27
Comment Exercise Assignments
28
Key Theme Ideas (please see page 34 in handbook)

• Tables share potential Key Theme topics
• Record potential key themes

29
Lessons Learned on Writing Process Comments
Scoring

• Open discussion from judges, overseers and all
  examiners
• What has worked well for you?
• Tip and tricks?

30
Producing Well-Written Results Comments (please
see page 24 in handbook)

• Results are outcomes
• Should link to requirements and measures
  identified in Process Categories and
  Organizational Profile
• Strengths should Identify where they fit in the
  Scoring Guidelines
• OFIs should identify what results will enable
  them to reach the next scoring box

31
Producing Well-Written Results Comments

• Comments should integrate results addressing
  common subjects. e.g. charts showing staff
  safety- back sprains, workers comp trends,
  lost-work days, etc.
• Use Figure numbers in the comment
• Should link to requirements and measures
  identified in Process Categories
• Strengths should Identify where they fit in the
  Scoring Guidelines
• OFIs should identify what results will enable
  them to reach the next scoring box

32
Results Pitfalls

• Repeating lack of comparative data at
  inappropriate scoring level
• Not sanity checking applicant charts
• Arrows going to wrong direction
• Providing an approach comment
• Providing so whats from an examiner perspective
  instead of applicant

33
Item Exercise

• Purposes
• To reinforce comment guidelines for the Results
  Category
• To identify opportunities for Examiners to
  improve their comment writing skills
• To calibrate Examine Scoring for Results

34
Exercise Requirements

• Exchange Case Study Designated Item with another
  teammate at your table
• Read and annotate your teammates comments
• Note well-written comments
• Provide suggestions of how to improve other
  comments
• Return annotated comments and discuss
• Compare scoring of all teammates at your table
• Discuss the scoring. Identify drivers of the
  score
• Report out on lessons learned

35
Exercise Activities
36
Results Comment Exercise
37
Key Theme Ideas (please see page 34 in handbook)

• Tables share potential Key Theme topics
• Record potential key themes

38
Lessons Learned on Writing Results Comments

• Open discussion from judges, overseers and all
  examiners
• What has worked well for you?
• Tip and tricks?

39
Preparing Key Themes

• Three dimensions
• Important areas of strength in Processes
• Key concerns or issues in Processes
• Key Strengths and concerns or issues in Results
• Limit to 3 -4 statements per dimension
• Each statement should be integration of findings
  from multiple Items
• Should be 2 -3 sentence narrative in terms of a
  finding

40
Key Themes

• Think of as an Executive Summary
• Not just a cut and paste of comments,
• but derived from multiple comments
• Should link to Criteria Core Values

41
Key Themes

• Should be the most important things for an
  organization to focus on after receiving the
  feedback
• Should aid the judges in determining if the
  organization has role model potential
• Will become key focus areas for site visit team
• Should relate to the Criteria Core Values and Key
  Business Factors

42
Consensus Scorebook
Score Sheet
KBFs
App Quality Team
Judges Decide On Site Visit Based On
43
Key Themes Exercise

• Purpose
• Enhance examiner ability to develop key themes

44
Exercise Requirements

• Identify Case Study key themes identified by
  tablemates
• Discuss differences in key themes for the
  assigned dimension
• Select one key theme in the assigned dimension
• Draft a Good key them for the assigned dimension

45
Key Theme Example

• The applicant demonstrates patient-focused
  excellence through multiple patient and other
  customer relationship approaches. These include
  the development of the CM health care delivery
  model and the use of multiple approaches to
  listen and learn from patients. In addition, the
  Partners Committee meets with senior leaders
  quarterly to discuss current and future patient
  needs.

46
Exercise Activities
47
Team Assignments
48
Expectations in Stage 1(please see page 36 in
handbook)

• Maintain confidentiality
• Open the Application immediately to check for
  Conflict of Interest
• Notify the Award Office when returning an
  unscored Application
• Allow adequate time to review an Application
• Expect 1 application
• Complete by 3 July

49
Stage 1 (please see page 35 in the handbook)

• All Examiners participate
• Assignment factors
• Maintain confidentiality
• Work independently
• Produce Scorebook
• Timeline
• Receive assignments in early June (after June
  6th)
• Three weeks to complete (June 7July 3)

50
Stage 1 - Prepare

• Open Application Package Immediately
• Check for conflict of interest
• Fax Conflict of Interest Statement to office
  [phone]
• Read entire application
• May want to list potential Key Themes
• Draft the Key Business Factors

51
Complete Scorebook
KBFs
Score Sheet
Who the organization is and what is important to
it
Strengths,next steps, and Score per Item
Score by Item
Most important overall strengths and next steps
52
Stage 1 Scorebook

• Send (WHOLE) Scorebook electronically to Award
  Office by July 3rd

53
Stage 1 - Highlights

• Independent Review - no teamwork
• COI - Check IMMEDIATELY
• Questions - call Award office [phone]
• Comments - Quality NOW
• Deadlines - must be met

54
Consensus - Overview

• All Examiners participate
• Work as a team
• Produce Consensus Scorebook

55
Consensus Objectives(please see page 42 in
handbook)

• Clarify and resolve individual differences
• Reach consensus on
• Item Comments
• Key Themes
• Scores
• Site Visit Issues
• Product is the Consensus Scorebook

56
Consensus Scorebook
Score Sheet
KBFs
Same deliverable as Stage 1
57
Consensus Process
Consensus Planning
Consensus Day
Post Consensus
58
Consensus Planning

• Award Office notifies Team leader of team
  membership
• Team leader notifies team members of team
  composition
• Team members exchange Stage 1 Scorebooks
  (electronically) and bios
• Team leader assigns Category Leads and Back-ups
• Team determines Consensus Day within 24 July - 4
  August window
• Award Office assigns an Overseer

59
Consensus Planning

• Team Leader (or as assigned) synthesizes KBFs and
  sends to all team members
• Category Leaders synthesize comments for the
  Items in their assigned Category
• Category Leaders rescore Items based on
  synthesized comments
• Category Leaders and back-ups exchange comments
  for polishing of comments and concurrence on
  scoring

60
Synthesizing Comments

• Read same Item for all Scorebooks
• Note areas of agreement and unique statements
• Note areas of disagreement accounting for scoring
  differences

Prepare
61
Synthesizing Comments

• Several Team members address
• same topic
• Craft comment that expresses
• content of the other team members
• Comments
• Consider KBFs
• Consider Criteria
• May use a particularly good comment
• as the starting point

Synthesis
62
Synthesizing Comments
Synthesis

• Only 1 Team member addresses
• a topic
• Determine validity by considering
• KBFs
• Criteria
• Level of Comment
• It is something missed by other
• team members
• If valid, Craft comment that expresses content of
  the members Comments
• If invalid, make note of why not used

63
Synthesizing Comments

• Select 4-6 Key Business Factors for each Item
• Propose 6-10 synthesized Comments for each Item
• Note members initials and Comment No. whose
  inputs were used
• Compile list of Comments not used by members and
  their Comment No.
• Score synthesized Comments

Write/Score
64
Synthesize Comments
Review

• Verify Comments cover main objectives of Criteria
• Verify Comments conform to comment guidelines
• Verify No Conflicts Between Strengths and OFIs
• Send to Back-up
• Revise Comments and/or
• score based on Back-up inputs

65
Other things in Preparation Phase

• Responsibility for Items in Category 7 will be
  distributed to Process Category Leads
• Allocate enough time to get Items synthesized and
  rescored
• A quality product makes the consensus day go much
  faster
• Keep in touch with your team leader

66
Final Preparation Checks
67
Synthesizing Exercise

• Purpose
• Gain experience in synthesizing comments

68
Exercise Requirements

• Split each table into two groups
• One group addresses strengths
• One group addresses OFIs
• With comments provided from the case study and
  your prework, each group synthesize comments
• Make list of not used comments with rationale
  for not using
• Record the synthesized comments
• Report out on one synthesized comment

69
Exercise Activities
70
Consensus Exercise Assignments
71
Lessons Learned Consensus

• Open discussion from judges, overseers and all
  examiners
• What has worked well for you?
• Tip and tricks?

72
Consensus Day(please see page 43 in handbook)

• Team Leader schedule Category sequence
• Category Leaders each present their Items and
  proposed score
• Bring copies to the meeting
• Team Members discuss
• No wordsmithing
• Look for conflicts with own Items
• Verify comments are central to the Criteria and
  relevant to KBFs
• Not used are not discussed unless a team member
  has questions

73
Overseer Role

• Ensure consistency of the process
• Final authority on Criteria meaning
• An expert available to help the team throughout
  the day
• Not a member of the team
• Has no vote

74
Consensus Day

• Scoring - STAGE ONE SCORES ARE DEAD! - what is
  the right score for the synthesized set of
  comments?
• Use Noodle Recharging Times (also known as
  lunch and breaks) wisely - dont discuss the
  applicant or process or try to conduct business
• Listen actively
• Arrive on time and plan to stay until the team
  has completed. Typically
• Start time 800 am
• Finish 600 pm - 800 pm

75
Guidelines For Resolution of Differences
Scoring Difference Among Examiners for an
Item Action 20 or Less Mean 30 or
Greater Confer
When computing the mean, round off to the nearest
whole number
76
Consensus Day

• Turn off cell phones
• Team Leader will provide an emergency number
• Come well-rested
• Dont just cave-in - you may be right!
• Leave knowing you have really helped the applicant

77
Post-Consensus

• Category Leaders smooth up Item Comments
• Send Team Leader finished Item worksheets
• Do the above as soon as possible while the
  discussions are fresh in your mind

78
Consensus Exercise

• Purpose
• To understand and practice reaching consensus on
  comments, and score

79
Consensus Exercise Requirements

• Each table will use comments from previous
  exercise
• Select Category Leader for Item to be discussed
• Read over the comments created during the last
  exercise
• Category Leader present comments and lead
  discussion
• Category Leader recommends a score
• Team comes to consensus on a score

80
Exercise Activities
81
Role of the Panel of Judges

• Judge Applicant Organizations
• Select for Site Visit
• Review Site Visit Reports and recommend Award
  applicants to Board of Overseers
• Improve assessment process
• Monitor and verify assessment process
• Recommend changes to Overseers

82
Judges Process

• Site Visit selection
• Analyze scores and scoring profiles from Stages 1
  and 2
• Receive input from Consensus Reports through
  Award Director and Overseer Chairperson
• Focus is on Key Themes
• No Conflict of Interest

83
Judges Process (cont.)

• Key Judge Discussion Areas
• Most important strengths or outstanding practices
• Most significant concerns, vulnerabilities or
  weaknesses
• Considering KBF's, what are the most significant
  strengths, vulnerabilities and/or gaps in Results
• Applicant performance relative to Core Values

84
Judges Approach

• Dont Block a Winner
• It aint over til its over - so dont assume
  your applicant wont get a site visit
• Is this organization a Missouri role model?

85
Next Steps

Stage 3 Site Visit September
Stage 2 Consensus Review July/August
Stage 1 Independent Review June/July

Feedback Report Completion August and October
86
After Consensus

• Judges select site visit applicants on August 16.
  Immediately thereafter
• Teams repotted
• Team leaders and applicants notified
• Team leaders contact team members and send
  consensus report to all
• Team leaders begin planning of site visit
• Site visit training on August 30
• Keep clear on your calendars!
• Will do team activities during the training and
  site visit planning

87
Site Visit Planning

• You will develop site visit issues for your
  assigned Categories
• Team Leader will have applicant validate KBFs
• Team Leader will coordinate planning with the
  applicant

88
Expectations in Stage 1

• Maintain confidentiality
• Open the Application immediately to check for
  Conflict of Interest
• Notify the Award Office when returning an
  unscored Application
• Allow adequate time to review an Application
• Expect 1 application
• Timeline June/July

89
Consensus Expectations

• No more than one Consensus assignment per
  Examiner
• Time commitment
• Hard work
• Confidentiality

90
Missouri Quality AwardExamination Process
June 1, 2006 Award Applications Received
June 7 -July 3, 2006 Examiners Score Applications
June 6, 2006 Applications Distributed to Examiners
July 3, 2006 Examiners Submit Scorebook to Award
Office
July 13, 2006 Senior Examiner Training
July 14-28, 2006 Consensus Preparation
July 24- August 4, 2006 Consensus Reviews
August 14, 2006 Consensus Results Due to Award
Office
August 16, 2006 Judges Review Consensus Results
to Determine Site Visit Applicants
Sept 10 - 29, 2006 Examiners Conduct Site Visits
August 30, 2006 Site Visit Training
October 17-18, 2006 Judges Final Review Make
Recommendations
October 19, 2006 MQA Board of Overseers
Determines Award Winners
November 16, 2006 MQA Conference Awards Banquet
91
Congratulations!
You are now a member Of the Board of Examiners!
92
Questions/Problems

• Contact
• Brenda Hatfield
• 573-526-1727
• Brenda.Hatfield_at_mqa.mo.gov
• or
• Katrina Prenger
• 573-526-1725
• Katrina.Prenger_at_mqa.mo.gov