The Secretarys Robert W' Carey Performance Excellence Awards

1
The Secretarys Robert W. CareyPerformance
Excellence Awards

• New Examiner Training
• May 19-21, 2009

2
Purpose of Course

• Provide participants with
• A working knowledge of the Carey Examination
  Process and the Baldrige Criteria for Performance
  Excellence
• The basic skills needed to examine and score an
  award application using the 2009 2010 Baldrige
  Criteria for Performance Excellence

3
Course Objectives

• At the conclusion of this training the
  participants will be able to
• Execute the Carey Examination Process and use the
  Baldrige Criteria for Performance Excellence
• Develop and use a Key Factors worksheet
• Draft appropriate strength and opportunity for
  improvement (OFI) comments
• Score an application using the Baldrige Criteria
  for Performance Excellence

4
General Information

• Emergency telephone number
• Messages
• Rest rooms
• Food
• Start/stop times
• Questions???

5
Ground Rules

• Respect the opinions of others
• Ask questions
• Remember what is said in class stays in class
• Be back from breaks on time
• Work hard and have FUN
• Others?

6
Introductions

• Pick a partner and gather the following
  information
• (Take approximately 5 minutes to talk)
• Name (Who are you?)
• Organization (Where do you work?)
• Job (What do you do?)
• Any Baldrige or Carey criteria experience?
• What is something others might not know about
  you?
• Introduce your partner to the class
• Take approximately 30 seconds to one-minute 30
  seconds

7
Carey Program Office and Co-Chairs

• Carey Program Office
• Scott Holliday, Director, Management Systems
  Improvement Service, VACO, 008B3
• Eric Malloy, Program Manager, Performance
  Excellence
• Gwendolyn Young, Diane Burton
• Carey Program Co-Chairs
• Michael Greenblatt, Office of Quality and
  Performance, VHA
• Michael Picerno, Calverton National Cemetery, NCA
• Gina Klaus, Office of Policy Program, VBA

8
Agenda Day 1

• Administrative Items, Pre-Test, and Introductions
• 2009 2010 Baldrige Performance Excellence
  Criteria
• Baldrige Systems Framework
• Core Values
• Sections of the Criteria book
• Review and discuss Criteria and corresponding
  responses in case study
• Carey Examination Process
• General examination process
• Examiner process for developing an application
  score book
• Discuss and develop a Key Factors Worksheet
  Case Study

9
Agenda Day 2

• Distribution of team information
• Introductions
• Review note-taking in the application
• ADLI
• LeTCI
• Writing Process Strength Comments
• Strength comment exercises
• Writing Process Opportunities for Improvement
  (OFI) comments
• OFI comment exercises

10
Agenda Day 2

• Writing Result Strength comments
• Strength comment exercises
• Writing Result OFI comments
• OFI comment exercises

11
Agenda Day 3

• Scoring Process
• Exercise scoring Process and Result Items
• Themes
• Completing a Scorebook
• Site Visits
• Post-Test
• End of Course Survey

12
Pre-Test

• Purpose To determine subject matter knowledge
  before in-class training begins
• Individual test scores will not be given to your
  supervisor
• 25 questions, 4 points each
• No penalty for guessing
• 15 minutes - maximum

13
(1) MSIS receives application
Carey Examination Process Overview
(2) Examiner team independent review
(3) Examiner team consensus
(4) Judge review Calibration
(5) Site Visit?
(5a) Finalize, Send to applicant
N
End
Y
(6) Clarify/verify information
(7) Other checks
(8a) Finalize, Send to applicant
(8) Array by score
N
End
Y
(9) Recommendations Sent to Secretary
14
Criteria are Built on a Set of Interrelated Core
Values and Concepts

• Visionary Leadership
• Customer or Patient-Focused Excellence
• Organizational and Personal Learning
• Valuing Workforce Members and Partners
• Agility
• Focus on the Future
• Managing for Innovation
• Management by Fact
• Societal Responsibility Community Health
• Focus on Results and Creating Value
• Systems Perspective

15
Key Sections of the Criteria for Performance
Excellence Book

• Review of Baldrige criteria book
• VA uses Business and Healthcare versions of the
  criteria. The books and the criteria are similar
  but page numbers vary
• Contents general flow of book
• Criteria
• About the criteria
• Scoring
• Application information (This section refers to
  the Baldrige national program. VA has its own
  application book with timelines and requirements)

16
Key Sections of the Criteria for Performance
Excellence Book

• Criteria for Performance Excellence
• Item listing shows point values for each Item and
  each category
• Turn ahead a few pages and look at 1 Leadership
• The criteria are located in the large shaded
  section. The information outside the shaded area
  describes the criteria and provides examples
• Key Characteristics of the Criteria and changes
  from 2008
• This section provides a brief summary of criteria
  changes for each Item

17
Key Sections of the Criteria for Performance
Excellence Book

• Criteria Response Guidelines
• The Criteria book is written for applicants, not
  examiners. This section provides a suggested
  process for preparing an application. The Carey
  has its own suggested process for developing an
  application
• Criteria Category and Item Descriptions
• This section provides a more detailed description
  of each criteria Item. It is an excellent
  reference source when you want a little more
  information than is offered in the foot notes

18
Key Sections of the Criteria for Performance
Excellence Book

• Criteria Core Values Concepts
• Values and concepts are embedded beliefs and
  behaviors found in high-performing organizations.
  While not specifically stated in the criteria,
  these values and concepts are the basis upon
  which the criteria are built.
• Glossary of Key Terms
• All the words in the criteria that are spelled in
  capital letters can be found in the glossary
• Scoring System Scoring Guidelines
• Describes the scoring aspects and contains the
  scoring guidelines

19
File 1 of 3
19
Version 1.0 May 2009
20
Levels of Maturity - Basic, Overall, Multiple
Requirements

• Basic Item requirements
• Responds to most basic or fundamental theme of
  the Item
• Overall Item requirements
• Responds to the central or most significant
  features or theme
• Multiple Item requirements
• Responds to the detailed requirements of the Item

20
21
Baldrige Framework A Systems Perspective
Organizational Profile Environment,
Relationships, and Challenges
2 Strategic Planning
5 Workforce Focus
I Leadership
7 Results
3 Customer Focus
6 Process Management
4 Measurement, Analysis, and Knowledge Management
22
Process Scoring Aspects ADLI

• Approach (A)
• Method used to accomplish a process or system
• Covers applicable aspects of criteria
• Approach is planned
• Method used is systematic or repeatable
• Deployment (D)
• Extent process is disseminated throughout the
  organization (depth and breadth)
• Approach is deployed consistently in all areas of
  the organization

File 1 of 3
22
23
Process Scoring Aspects ADLI

• Learning (L)
• Cycles of evaluation and improvement
• Encourages breakthrough change to approaches
• Sharing, adapting and adopting refinements, and
  innovations within the organization
• Integration (I)
• Approach aligned with organizational needs
• Approach integrated between organizational units
• Harmonization of plans, processes, and results to
  support organization-wide goals
• Individual components operate as a fully
  interconnected unit

File 1 of 3
23
24
Results Scoring Aspects - LeTCI

• Levels refer to the applicants current level of
  performance
• How good are the applicants results?
• Are performance levels reported on a scale that
  is meaningful to the applicant?
• Trends refer to rate of performance improvement,
  sustainability of good performance, or breadth of
  performance results
• A minimum of three historical (not projected)
  data points makes a trend

25
Results Scoring Aspects - LeTCI

• Comparisons refer to performance relative to
  appropriate competitors, peers, industry leaders
  and/or benchmarks
• Integration refers to result measures (often
  through segmentation) to important customer,
  product and service, market, process, and action
  plan performance requirements identified in its
  Organizational Profile and in Process Items

26
Criteria Items and Case Study Responses

• Item 1.1 Senior Leadership
• Vision, Values, and Mission
• Case study - an example of APPROACH
• Page 2 describes setting and deploying mission
  and values as part of planning process (figure
  2.1-1) and communication methods (figure 1.2-1)
• Communication and Organizational Performance
• Case Study an example of LEARNING
• Page 3, .b.1 Improvements implemented in 2009
  cycle

27
Criteria Items and Case Study Responses

• Item 1.2 Governance and Societal Responsibilities
• Organizational Governance
• Case Study an example of DEPLOYMENT
• Page 5, Figure 1.2-1 Key Aspects of Governance
  System show deployment to the Carey team, not
  clear if volunteers are included in these
  descriptions
• Legal and Ethical Behavior
• Discuss
• Societal Responsibilities and Support of Key
  Communities
• Case Study- an example of INTEGRATION
• Page 7, c.2 Figure 1.2-4 shows alignment of key
  communities with vision

28
Criteria Items and Case Study Responses

• Item 7.6 Leadership Outcomes
• Leadership and Societal Responsibility Results
• Case Study An example of LEVELS
• Page 49, Regulatory and accreditations all
  current or in compliance so levels appear to be
  good
• Case Study An example of COMPARISONS
• Page 48, Few Comparisons provided just figure
  7.6-1

29
Criteria Items and Case Study Responses

• Exercise 25 minutes
• In this exercise we will review the remaining
  criteria Items and find examples of the scoring
  aspects from the case study so you can become
  familiar with the types of responses to look for
  in an application.
• Table assignments
• Table 1 Items 2.1, 2.2, 7.1
• Table 2 Items 3.1, 3.2, 7.2
• Table 3 Items 4.1, 4.2, 7.3
• Table 4 Items 5.1, 5.2, 7.4
• Table 5 Items 6.1, 6.2, 7.5

30
Criteria Items and Case Study Responses

• Exercise contd
• For assigned Items
• Read the Item Criteria
• Review the responses in the case study
• For the first Item in a category, Identify an
  APPROACH and a LEARNING
• For the second Item in a category, identify
  DEPLOYMENT and INTEGRATION
• For the Result Item identify any two of the four
  scoring aspects LeTCI
• Identify a team presenter

31
Criteria Items and Case Study Responses

• Exercise contd Discussion
• Read the Process Item Overall Requirements
• Identify the page and location of the two
  examples identified in the case study
• Identify case study examples as A, D, L, or I
• Repeat for each Process Item
• Read the Result Item Overall Requirements
• Identify the page and location of the two
  examples identified in the case study
• Identify the two aspects evaluated and describe
  performance in these aspects

32
(1) MSIS receives application
Carey Examination Process Overview
(2) Examiner team independent review
(3) Examiner team consensus
(4) Judge review Calibration
(5) Site Visit?
(5a) Finalize, Send to applicant
N
End
Y
(6) Clarify/verify information
(7) Other checks
(8a) Finalize, Send to applicant
(8) Array by score
N
End
Y
(9) Recommendations Sent to Secretary
33
Key Examiner Products

• Conflict of Interest and Confidentiality forms
• Examiner analysis consisting of
• Scorebook
• Key factors worksheet
• Themes
• Executive Summary
• Strengths and Opportunities for Improvement for
  each Item (4 to 8 comments per Item)
• Scores
• Individual Score Sheets
• Team Score Sheet

34
Check for Conflict of Interest
Review Criteria And Read Application
Draft Key Factors
Consensus on key Factors
Take notes on application
Finalize comments
Independently score Items
Draft Comments for assigned Items
Consensus on scores
Complete Scorebook
Identify Key Themes
Consensus on comments
Examination - Scorebook Development Process
35
Check for Conflict of Interest
Review Criteria And Read Application
Draft Key Factors
Consensus on key Factors
Take notes on application
Finalize comments
Independently score Items
Draft comments for assigned Items
Consensus on scores
Complete Scorebook
Identify Key Themes
Consensus on comments
Examination - Scorebook Development Process
36
Conflicts of Interest/ Confidentiality Statement

• Tab E forms
• Affirm that you dont have any conflicts of
  interest
• If you think that there may be a conflict of
  interest call Eric Malloy, Carey Program Manager
  at 202-461-5771
• If there is a conflict, you will be assigned to
  another application

37
Conflicts of Interest/ Confidentiality Statement

• Do not disclose any information about the
  applicant gained through the evaluation process
  to anyone outside the applicant review process

38
Review Criteria And Read Application
Check for Conflict of Interest
Draft Key Factors
Consensus on key Factors
Take notes on application
Finalize comments
Independently score Items
Draft comments for assigned Items
Consensus on scores
Complete Scorebook
Identify Key Themes
Consensus on comments
Examination - Scorebook Development Process
39
Read the Application

• Quickly read the application from cover-to-cover
• As you read the application a second time, mark
  it up and put notes in the margins
• Identify linkages
• Organizational Profile to processes
• Processes to Results
• Organizational Profile to Results

40
Check for Conflict of Interest
Review Criteria And Read Application
Draft Key Factors
Consensus on Key Factors
Take notes on application
Finalize comments
Independently score Items
Draft comments for assigned Items
Consensus on scores
Complete Scorebook
Identify Key Themes
Consensus on comments
Examination - Scorebook Development Process
41
Organizational Profile

• Snapshot of the applicants organization
• Sets Context for later Items
• Two Parts
• Organizational Description
• Organizational Challenges
• Starting point in understanding whats important
  to the applicant organization
• Helps identify key gaps in information,
  requirements, or results
• Administratively used by Examiners or Judges
• Understand the organization
• Understand the basis of comments (/-)

42
Key Factors (KF) Worksheet

• A Key factors Worksheet is a list of key points
  primarily from the Organizational Profile
• Worksheet organized into the following five
  sections
• Organizational Environment
• Organizational Relationships
• Competitive Environment
• Strategic Context
• Performance Improvement System

File 1 of 3
42
43
Key Factors are used to examine each Item

• Identify KFs that are associated with Criteria
  Item being worked
• Evaluate criteria responses using KFs as a tool
  for understanding how criteria relate to
  applicants organization
• KEY POINT As an examiner, you want to look for
  linkages to KFs and make sure responses address
  them
• Since criteria are generic, KFs relate criteria
  to applicants organization
• For example, if KFs identify three customer
  segments then Customer Focus responses should
  discuss those three segments. Customer Focused
  Results should also address the requirements of
  these same three customer segments. Links to
  other criteria responses may also be found.

43
44
Developing a Key Factors Work Sheet

• Exercise 20 minutes
• Read assigned area of the Organizational Profile
  from the Criteria
• Read case study responses to assigned area of the
  Organizational Profile
• Identify Key Factors and record information on
  flipchart paper
• Identify a team presenter

44
45
Summary of Day 1

• Baldrige Framework
• Reviewed all Criteria Items and associated
  sections of the case study
• Conflict of Interest and Confidentiality forms
• Carey Award and Examination Scorebook Development
  Processes
• Organizational Profile and Key Factors

File 1 of 3
45