Title: Examiner Training
1Examiner Training
- Process comments
- Results comments
- Scoring
2Agenda
- Classroom training (Examination focus) Day 3
- Key Factors
- Process Items
- Strength Comments
- Opportunity for Improvement Comments
- Results Items
- Strength Comments
- Opportunity for Improvement Comments
3Agenda
- Classroom Training (Examination focus) Day 4
- Scoring process for Process and Results Items
- Overview of what to expect as a member of the
Board of Examiners - Graduation
4Agenda
- Classroom Training (Examination focus) Day 4
- Scoring process for Process and Results Items
- Overview of what to expect as a member of the
Board of Examiners - Graduation
5General information
- There were two trophy winners in 2009. The
winners were Bay Pines VA Healthcare System and
the Louisville VA Medical Center. The Bay Pines
2009 application will be used as the case study
for this section of the training. - Sections of the Bay Pines VAMC application to be
used in training include - Organizational Profile
- Responses to Items 1.2, 2.1, and 7.6
6What have we already learned?
- A great deal of the information examiners need to
know has already been covered in this class.
This includes - Understanding Approach, Deployment, Learning, and
Integration - Understanding Levels, Trends, Comparisons, and
Integration - Developing and using a Key Factors Worksheet
7We also learned
- The following are NOT considered when assessing
an application - Organization of the application. As an examiner,
if you can find the information needed then you
give the applicant credit - Typos
- Incorrect figure numbers
- Poor quality binding
8General Examination Process
- Review application to determine if you have a
conflict of interest - Conflicts include
- You examined this applicant within the past three
years - You worked for this applicant within the past
three years - Your close friend or relative either works at the
applicant site or is a customer of the site. - This includes the recent past. For example, my
dad was a patient at X. He past away two
months ago. - NOT a conflict
- You examined the applicant five years ago
- You worked for the applicant 8 years ago
- You participated in a SOARS or OAI site visit to
the applicant last year
9General Examination Process
- Read through the entire application (a quick
read) - Develop a Key Factors Worksheet
- Associate Key Factors with a Criteria Item
- Examine the Item responses and compare to the
Criteria - Take notes in the application book
- Draft Strength and OFI comments
- Repeat steps 3 through 5 for all Items
- Score all Items
10Developing a Key Factors Worksheet
- Read Organizational Profile questions in the
Criteria book - Review Organizational Profile responses in the
application and underline key facts - Draft a Key Factors Worksheet by recording the
key facts about the applicant - This Key Factors Worksheet is a key tool in
examining an application
11Developing a Key Factors Worksheet
- Exercise 20 minutes
- Review Organizational Profile responses in the
application - Underline key facts
- We will use this marked-up Organizational Profile
as our Key Factors Worksheet for purposes of the
training class
12Associate Key Factors with an Applicants Item
Responses
- Exercise 10 minutes
- Read the Overall Item Requirements for 1.2
- Review the applicants Organizational Profile
- Select the 4 6 underlined facts that you
believe are relevant when evaluating the
applicants responses to this Item - Record these facts at the topic of a piece of
paper
13Taking Notes in an Application
- Exercise 20 minutes
- Evaluate 1.2b(12) and 1.2c(12)
- Read the applicants responses to Item 1.2
- For each Multiple Requirement within the Criteria
- Read the criteria question
- Read the applicants response
- Write A or A-, D or D-, L, or L- in the margin
next to the applicants response to indicate if
the scoring factor was positively addressed or not
14Process Strength Comments
- Strength comments
- Are used when the applicant provides evidence of
ADLI in reference to the criteria - Strength comments contain two parts
- An initial phase or sentence linking the strength
to the criteria - A final phrase or sentence providing evidence
(examples) from the application to support the
strength
15Process Strength Comments, Part One
- Verb(s) and specific Requirements from the
Criteria - review organizational performance
- Communicates, listens, and learns
- Makes needed data and information available
- Subject from the Criteria or application
- The senior leaders
- The organization
- The school
- The medical center
- The cemetery
- The applicant
16Process Strength Comments, Part One
- EXAMPLE
- 1.1.a(1) Senior leaders set and deploy the
organizations vision and values through the
leadership system to the workforce... - Note When you pull from the Criteria to write
your comment, you must be accurate. The Criteria
says to the workforce, to key suppliers and
partners, to patients and to other customers.
In this case there is the link to the Criteria
and that is what is evidenced in the application.
17Process Strength Comments, Part One
- EXAMPLE
- 1.1.a(1) Senior leaders set and deploy the
organizations vision and values through the
leadership system to the workforce... - Note
- Address Item 1.1, Area a, multiple
requirement (1) - Question Which aspects are addressed in this
comment? A, D, L, and/or I
18Process Strength Comments, Part Two
- Example(s) from the application
- Provide the activities or processes that led you
to the conclusion that this was a strength the
facts, the WHAT, the HOW
- Transition
- By/through/using
- This is demonstrated by
19Process Strength Comments, Part Two
- by using an annual Strategic Planning process
and numerous communication methods, such as news
letters, meetings, and stakeholder sessions to
communicate to all faculty, staff, students, and
key stakeholders and partners. The leadership
system involves many participants and
decision-making levels. - Note This information comes from the
application.
20Process Strength Comment - Tips
- Your comment can be about one or more of the four
aspects (ADLI) - Only comment on the aspects that are particularly
strong - Support your findings with examples from the
application - Review the scoring guidelines does your comment
align with the performance level that you feel is
representative of the applicant?
21Process Strength Comments
- Exercise 30 minutes
-
- Review your 1.2b(12) notes
- Draft two Strength comments
- For each comment
- Underline the link to the criteria
- Circle the examples from the application
- Identify a speaker for the group
22Process Opportunities for Improvement (OFI)
Comments
- The second part of an Item level examination is
the identification and preparation of OFIs - An OFI identifies an area (usually one factor) in
the applicants response that in some way does
not fully respond to a criteria question and may
be of value to the applicant in its efforts to
achieve its business objectives - How does an examiner determine if a possible OFI
is of value to the applicant?
23Process Opportunities for Improvement Comments
- OFIs contain two parts
- A statement containing the Criteria and examples
from the application - A statement describing the possible impact on the
organization
24Process Opportunities for Improvement Comments
Part One
- Subject from the Criteria or Application
- The senior leaders
- The organizations
- The medical center
- The MSN
- The regional office
- Introductory Statement
- It is not clear how/what
- It is unclear how/what
- It is not evident how/what
- It is not apparent how/what
25Process Opportunities for Improvement Comments
Part One
- Verb(s) and specific requirements from the
Criteria - Review organizational performance
- Communicates, listens, and learns
- Makes needed data and information available
- Complaint management process
26Process Opportunities for Improvement Comments
Part Two
- The second part of an Opportunity for Improvement
comment is a statement describing the possible
impact on the organization. This is often
referred to as a so what statement - Why dont we use so what statements in Strength
comments? - Why do we use so what statements in an
Opportunity for Improvement comments? -
27Process Opportunities for Improvement Comments
Part Two
- The so what statement describes the benefit or
consequence associated with the Opportunity for
Improvement. - If you know the part of the criteria that is not
addressed in the applicants response you could
look up a standard OFI response in a Baldrige
reference book - For example, an applicant does not gather
information from customers of competitors. The
so what might be, - Without information from customers of competitors
leaders may be hindered in their ability to win
over these customers and gain market share.
28Process Opportunities for Improvement Comments
Part Two
- The applicant puts a great deal of time and
effort into their application - Examiners want to provide value-added information
to the applicant - Value-added OFIs are linked to areas that have
been identified as important to the applicant - So what statements need to be specific enough
and important enough to motivate applicant
leadership to act - A generic OFI from a reference book may be a
start but it is not specific enough to provide
the level of importance needed to facilitate
action - ALSO, MOST APPLICANTS ARE SMART ENOUGH TO
IDENTIFY GENERAL OFI COMMENTS
29Process Opportunities for Improvement Comments
Part Two
- So, where does the information to support a good
so what statement come from? - Answer The Organizational Profile - the Key
Factors Worksheet -
30Process Opportunities for Improvement Comments
Part Two
- Revised OFI comment
- Without information from customers of competitors
leaders may be hindered in their ability to
identify new products and services that may be
critical keeping current customers that have a
choice in health care providers.
31Examining an Application
- Exercise 20 minutes
- Review the criteria, the applicants response,
and your notes for Item 1.2.c(12) - Identify one potential OFI
- Draft an OFI comment
- Use the standard two-part comment structure
- Underline the first part of the comment
- Circle the so what portion of the OFI
- Identify a speaker for the group
32Process Examination
- Exercise 30 minutes
- Read Criteria Overall Requirements for Item 2.1
- Select 4 6 Key Factors
- Take notes on 2.1a(1)
- Draft 1 Strength comment
- Draft one OFI comment
33Results Examination
- Examining the Results Items
- Similar to evaluating process Items except the
scoring factors change. The examination scoring
factors are - Levels (Le)
- Trends (T)
- Comparisons (C)
- Integration (I)
34Example showing Level
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
35Example showing Levels with Segmentation
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
36Example showing Levels, Segments, and Trends
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
37Example showing Levels, Segments, Trends, and
Comparisons
Benchmark
Satisfied or Extremely Satisfied
Competitor
Industry Average
Figure 7.1-1 Customer Satisfaction with Product
Quality
38Category 7Example of Strength Comment
- Class Exercise
- What scoring aspects are addressed in this
strength comment? -
- 7.6a(3) Over the last three years, the
applicant has increased the percentage of
external board members from 25 to 60 (Figure
6.7-1) and has appointed an external director as
the head of its audit committee. These results
are related to the applicants strategy of
achieving greater independence in oversight of
governance and financial audits.
39Results Strength Comments
- Results Strength comments are similar in
structure to Process Strength comments. - There are two parts to a Results Strength
comment. - The first part references the scoring aspect that
is the focus of the comment (Levels, trends,
Comparisons, and/or Integration). - The second part of the comment provides examples
(evidence) from the application to support the
finding.
40Category 7Examination of Results
- The keys to successful examination of Items in
Category 7 are - Identifying Expected Results and Key factors
- Assessing performance levels in relation to
scoring aspects - What results do you expect to find in Category 7?
- For results shown, assess in terms of (Le, T, C,
and I)
41Category 7Results Comments
- Comment Structure
- Elements subject (applicant or result),
timeframe (in, to/from), numbers (be specific),
figure reference number (always cited), and so
what statement (if applicable) - All key areas of the Item responses should be
reflected in the Item feedback comments - Scoring Aspects all scoring aspects (Le, T, C,
I) should be addressed in comments
42Category 7Example of OFI Comment
- Exercise
- What scoring aspects are addressed in this OFI
comment? - Although overall satisfaction levels are shown,
no segmented results are provided. The applicant
identified inpatient, elderly women, and
long-term care as key customer segments. Without
results for these segments it may be difficult
for leaders to assess progress in meeting the
unique needs identified as important to
satisfying these customers and meeting business
objectives.
43Category 7Examination of Results
- The keys to successful examination of Items in
Category 7 are - Identifying Expected Results and Key factors
- Assessing performance levels in relation to
scoring aspects. - What results do you expect to find in Category 7?
- For results shown, assess in terms of (Le, T, C,
I, and Gaps)
44Category 7Example of Strength Comment
- Exercise
- What scoring aspects are addressed in this
strength comment? - Overall satisfaction levels are consistently
higher than VHA average in each of the three key
customer requirements (access, courtesy, and
quality of care) as shown in Figure 7.2-3. The
Overall Customer Satisfaction Access levels are
in the mid-90s for the past three years,
consistently above VHA average (see Figure
7.2-4). The Overall Customer Satisfaction
Courtesy levels show an increase from 80 in 2003
to 88 in 2005 (see Figure 7.2-5). The Overall
Customer Satisfaction Quality of Care levels
are sustained at high levels of satisfaction, 97
each of the past three years (see Figure 7.2-6).
This places the applicant in the top 10 VHA
facilities each year for quality of care.
45Category 7Example of OFI Comment
- Exercise
- What scoring aspects are addressed in this OFI
comment? - 7.1a While the applicant shows overall
customer satisfaction with product quality for
2002, no trend data are presented to allow it to
track the rate of improvement (Figure 7.1-1). In
addition, the data are not segmented by key
customer group (i.e., industrial or commercial).
Thus, it is not clear how the applicant
identifies variation in its customer satisfaction
over time or from sector to sector to target
opportunities for improvement.
46Category 7Example of OFI Comment
- Exercise
- What scoring aspects are addressed in this OFI
comment? - Although overall satisfaction levels are shown,
no segmented results are provided. The applicant
identified inpatient, elderly women, and
long-term care as key customer segments. Without
results for these segments it may be difficult
for leaders to assess progress in meeting the
unique needs identified as important to
satisfying these customers and meeting business
objectives.
47Results Comments
- Exercise 20 minutes
- Review Overall Requirements for Item 7.6
- Identify Expected Results
- Identify key Factors
- Evaluate information provided using Le, T, C, and
I - Draft one Strength comment
- Draft one OFI comment
48Scoring Items
- Scoring can easy or difficult
- If you have well-written comments that cover the
Item criteria and areas most important to the
applicant, then scoring is easy. - If comments are general focused on a narrow
portion of the Item Criteria concentrated on one
or two examination aspects and not aligned with
areas of importance to the applicant then scoring
is difficult.
49Scoring Process
- Individual
- Review all comments (Strengths and Opportunities
for Improvement) associated with an Item - Turn to the appropriate set of scoring guidelines
in the Baldrige criteria book or use scoring
sheets - Begin with Approach (A) at the 50 65 score
range - Read the score range description
50Scoring Process
- Individual
- Is this description consistent with comments in
your analysis of the Item? - If yes, read the range above and below to ensure
you are comfortable with the range selected.
Indicate if the applicant is in the high, middle,
or low end of the selected range. - If no, move up or down until you read a range
that is consistent with your Item comments.
Then, read the range above and below to ensure
you are comfortable with the range selected.
Indicate if applicant is in the high, middle, or
low end of the selected range.
51Scoring Process
- Individual
- Repeat this process for each of the scoring
aspects (A, D, L, I for Process Items and Le, T,
C, I or Results Items). - Once score ranges have been selected for all four
scoring aspects, select an overall score range
that you believe represents the applicants
demonstrated performance in this area. - Review comments and scores.
- Select a percent score (score in increments of
5). For example 30, 40, or 45.
52Scoring Process
- Individual
- Repeat this process for each Item
- Team
- Place Item level scores in a table displaying the
scores of everyone on the team. - Calculate/estimate the average score for the
Item. - If individual scores are within plus or minus 10
of the average, average all scores and move on to
the next Item.
53Scoring Process
- If scores vary by more than 10 from the average
score the team needs to discuss, understand, and
reach consensus - Have the person with high score present facts
supporting the higher score (linked to the four
scoring aspects) - Have the person with the low score present facts
supporting the lower score (linked to the four
scoring aspects) - The entire team listens, considers, and rescores
54Scoring Process
- Things to consider
- 10 averaged with 4 other people on the team
amounts to about a 2 change in score. - 2 of an Item is about 1.5 points.
- There are a total of 1000 possible points.
- How long do you want to spend trying to reach
consensus on 2 points? - Carey specific procedure
- If individual scores are more than 15 apart for
any individual Item, that Item will be flagged
for the judge. A detailed review of the
applicant responses, team comments, and score
will be conducted and the judge will determine
what changes are needed.
55Scoring
- Exercise Part I 15 minutes
- Case study comments for Item 1.1
- Review all comments for Item 1.1
- Individually score using process discussed
- Share individual scores with people at your table
- Record individual scores
- Discuss
- Rescore
- Record consensus score
56Scoring
- Exercise Part II 15 minutes
- Case study comments for Item 7.6
- Review all comments for Item 7.6
- Individually score using process discussed
- Share individual scores with people at your table
- Record individual scores
- Discuss
- Rescore
- Record consensus score
57Carey Examination Process Flow
- Examiners
- Selected and trained
- Assigned to teams (4-7 on a team)
- Receive application, conflict of interest form,
reporting instructions in the mail - 2 3 days of independent work prior to team work
activities face-to-face - Read application
- Key Factors Worksheet
- Notes in Application
- Draft comments for assigned Items
58Carey Examination Process Flow
- 3.5 days in Washington area for face-to-face
meeting - Examine application - consensus on comments and
scores - Produce a draft feedback report
- Report to Carey process oversight group (program
manager and representative from each of the three
administrations) - Oversight group reviews all work and decides
- Which applications are recommended for site visit
- Which applications are not recommended for site
visit - Which teams may have had an issue during the week
- Which teams did an unacceptable job and require
another examination - EVERY MEMBER IS EMPOWERED TO FLAG AN APPLICATION
FOR ANOTHER REVIEW IF NEEDED
59Carey Examination Process Flow
- All applications and team analyses sent to an
objective, external, Baldrige expert, serving as
a Carey judge. Judges review the feedback report
and scores. Judges then calibrate scores and
comments to ensure a fair and objective
assessment of each applicant - Site visit conducted at facilities that have the
potential of being recommended for the trophy
award - All applicants scoring over 330 are evaluated to
ensure that there are no issues that might
otherwise exclude them from being recognized by
the Secretary. This could eliminate a fair
number of potential award recipients - Feedback reports prepared and sent to applicant
organizations
60Examining Process Flow
- A list of those remaining applicants with scores
over 330 is sent to the Secretary with
performance (score information), a summary of
applicant best practices, and general
recommendations (330 430 Achievement, 431 and
over Excellence or Trophy). - Recommendations are considered, then the
Secretary selects Achievement, Excellence, and
Trophy winners - Symposium and Awards Ceremony in fall
61Post test
- Please take 5 - 10 minutes to complete the post
test. - Review and correct in class
62THANK YOU