Welcome, Examiners

1
Welcome, Examiners!
Washington State Quality Award Return Examiner
Training 2009
2
First things first!

• THANK YOU!
• WSQAs mission Improving the way we live, learn,
  and work in Washington by helping organizations
  improve through the use of the Baldrige Criteria
  for Performance Excellence

3
Objectives For This Class

• Review Changes to the Baldrige Criteria
• Strengthen Skills in Areas of Opportunity
• Review Resources for Examiner Software

4
Agenda

• WSQA organization overview
• Baldrige criteria changes
• WSQA ethics review
• WSQA process Full Vs Lite
• Review process
• Individual review worksheets
• Consensus process
• Improvements
• Examiner Software
• Summary

5
For a Successful Course

• Be an active participant
• Give only constructive feedback
• Be courteous to your colleagues
• Honor diverse opinions
• Seek to understand
• Ask questions
• Have fun!!

6
Introductions

• Name, organization, what you do
• Questions and expectations?
• What do you like to do when youre not working?

7
What is WSQA?

• Created by State law to
• Promote excellence
• Recognize achievement
• Educate and train
• Patterned after the Baldrige National Quality
  Award
• Awards presented annually by the Governor
• Awarded to private, public, and not-for-profit
  organizations in manufacturing, service,
  education, and healthcare

8
What is WSQA?

• WSQA is a 501(c)(3) nonprofit organization
  supported by
• Corporate donations
• Individual memberships
• Workshop, collaborative and conference fees
• Many (many, many!) volunteer hours

9
Who is WSQA?

• Examiners
• Lite Assessment
• Full Assessment
• Certification follows assessment
• Senior Examiners
• Level 3 Team Leads
• Scorebook Editors
• Panel of Examiner Process Development (PEPD)
  Mentors
• Panel of Judges
• Board of Directors
• Executive Director and support team

10
2. Baldrige criteria changes

11
Baldrige criteria booklet

• Beginning in 2009 revised every 2 years

12
Baldrige 2007 Criteria Changes

• Strategic advantages
• Workforce engagement
• Leadership- performance measures
• Strategic Planning-innovation, strategic
  advantages

13
Baldrige 2008 Criteria Changes

• Order of information in the booklet
• Scoring system description
• Diagrams added for core values and maturity
  levels
• Added strategic advantages as a term to the
  Glossary

14
2009 Criteria Changes

• Customer Focus/ Engagement
• Organizational Core Competencies
• Sustainability and Social Responsibilities

15
Customer Focus/ Engagement

• Customer commitment to your brand and offerings
• Characterized by loyalty and willingness to make
  an effort to do business with you, willingness to
  do business with you

16
Core Competencies

• Areas of greatest expertise
• Strategically important
• Provide market place advantage
• May involve technologies, unique offerings,
  market place niche, business acumen

17
Sustainability

• Organizations ability to address business needs
  and agility and strategy to address the future
• Considers external and internal factors
• Considers impact on society and well being of
  environment, social and economic systems

18
Conflict of interest issues

• WSQA seeks to avoid conflict of interest to
  protect process integrity
• Conflicts are frequent!
• First step in assignment process determine any
  conflicts

19
Code of Ethical Conduct

• Why are ethics of utmost importance to WSQA and
  this process?

20
4. WSQA Full and Lite Process

• Comparison table

21
5. WSQA Process

Objective Review key steps in the process.
Review key process opportunities.
22
Stages of the Assessment Process
23
Preparing for independent review

• Read the criteria
• Read the application
• Familiarize yourself with the online tool
• All information on the WSQA website webinars and
  manuals http//www.wsqa.net/extimetable.php
• Check in with Team Lead on Key Factors and after
  entering category 1 and associated results

24
Independent review

• Select the most relevant Key Factors (from list)
• Read the criteria for the category
• Read (and annotate) the relevant section of the
  application
• Identify and record the applicants processes
• Record observations. Note missing responses and
  significant observations
• Recommend a scoring range for the category

25
Independent review steps - process

• Review related criteria
• Read the category under review, take notes
• Select relevant key factors from list
• Identify record processes
• Record observations on ADLI
• Approach
• Deployment
• Learning
• Integration
• Identify any significant or missing responses

26
Record observations on ADLI

• Approach refers to
• Methods used to accomplish the process
• How appropriate the methods are to the item
  requirements
• The effectiveness of their use
• The degree to which the process is repeatable and
  is based on reliable data and information (i.e.,
  systematic)

27
Record observations on ADLI

• Deployment refers to the extent to which
• The approach is applied in addressing relevant
  and important item requirements
• It is applied consistently
• It is used by all appropriate work units

28
Record observations on ADLI

• Learning refers to
• Refining the approach through cycles of
  evaluation and improvement
• Encouraging breakthrough change to your approach
  (innovation)
• Sharing refinements and innovations with other
  relevant work units and processes

29
Record observations on ADLI

• Integration refers to the harmonization of
• Plans
• Processes
• Information
• Resource decisions
• Actions
• Results
• Analyses
• to support key, organization-wide goals

30
Record observations on ADLI

• Questions to ask in analyzing Integration
• Do the individual components of a performance
  management system operate as a fully
  interconnected unit?
• Is the approach aligned with the organizational
  needs identified in the Organizational Profile?

31
Process diagnosis key concepts

• Integration examples
• Alignment of objectives and action plans with
  strategic challenges AND mission, vision, values
• Alignment of product/service delivery methods
  with KEY customer requirements
• Alignment of key process measures with KEY
  customer requirements

32
Process diagnosis key concepts
1. Approach
4. Integration
2. Deployment
PROCESS
3. Learning
33
Building the ADLI worksheet
How would you score this single response?
34
Independent review steps - results

• Review the Criteria to identify required results
• Review the relevant process in the application to
  identify KEY results important to the applicant
  (use KFs, application, your diagnosis)
• Review the Results in the application
• What IS there?
• What ISNT there that should be?
• Evaluate score on the basis of LeTCI

35
Results evaluation key concepts

• Levels
• Numbers that position results on a meaningful
  measurement scale
• Permit evaluation relative to
• Past performance
• Projections
• Goals
• Appropriate comparisons

36
Results evaluation key concepts

• Trends
• Numbers that indicate direction and rate of
  change
• Provide a time sequence of performance
• Require at least three data points
• Criteria call for trend data on
• Product, service performance
• Customer, workforce satisfaction
• Financial performance
• Market performance
• Operational performance (cycle time,
  productivity)

37
Results evaluation key concepts

• Comparisons
• External requirements
• External benchmarks
• Similar OR different organizations with similar
  characteristics

38
Results evaluation key concepts

• Integration
• Results align with key factors, e.g.,
• Strategic challenges,
• Workforce requirements
• Vision, mission, values
• Results presented for
• Key processes
• Key products, services
• Strategic accomplishments
• What examples can you think of?
• Strong integration?
• Not-so-strong integration?

39
Results evaluation key concepts

• Strong integration results presented for
• Key areas addressing strategic challenges,
• Key competitive advantages
• Key customer requirements
• Not-so-strong integration
• Results missing for the above
• Results presented that the Examiner cant match
  to process items or Organizational Profile

40
Key results

• Refers to elements or factors most critical to
  achieving the intended outcome
• In terms of results, look for
• Those responsive to the Criteria requirements
• Those most important to the organizations
  success
• Those results that are essential elements for the
  organization to pursue or monitor in order to
  achieve its desired outcome

41
Break time!
42
Stages of the Assessment Process
43
WSQA Consensus Step-by-step
44
WSQA Consensus Step-by-step
45
WSQA Consensus Step-by-step
46

• Stage 2 Consensus
• Webinar
• http//www.wsqa.net/extimetable.php
• Comments from all Independent reviews
• Ensure comments reflect scoring. - what is
  preventing applicant from moving to next level?
  Is this explained in the comments

47
Consensus preparation and call(s)

• Each Examiner responds with agreement,
  suggestions for changes, or disagreement, stating
  rationales
• Category Lead facilitates consensus
• Team Lead and/or PEPD mentor leads final scoring
  discussion
• Can take anywhere from 4-8 hours (depending on
  what?)

48
Consensus and conclusion

• If using online tool during consensus, changes
  may be made in real time
• If not, Category Leads update comments scores
  after call according to consensus decisions
• Category Lead notifies Team Lead and Scorebook
  Editor when done
• Scorebook Editor compiles consensus scorebook
  with final edits, notifies WSQA
• Consensus scorebook should be finalized ASAP

49
Comment, scoring considerations

• Benefit of the doubt
• Give credit for whats in the application
• Dont penalize for incidental exceptions
• Not every process must show complete DLI
• Not every results example must include TCI

50
Comment, scoring considerations

• For LITE, only one score for entire results
  Category 7
• Propose a score for results related to your
  categories
• Be prepared to reach team consensus for all of
  Category 7
• Using scoring language can help the applicant
  understand comments and score

51
Opportunities

• Examiner training
• Completing the stage 1 grid
• Using the gaps and notes tab
• Comment writing
• Consensus Meetings
• Scoring consistency
• Integration
• Learning
• Comparative Data
• Considerations for small organizations

52
National Examiner Questions2009Harry
Hertz Responses
Baldrige National Quality Program
53
Question 1

• Further clarification of performance
  projections vs goals
• Inclusion of goals in results category why are
  they not included in the evaluation?
• Why are projections expected for all measures in
  all scoring ranges?
• Why are projections not in the Results Criteria?

54
Question 2

• Why dont we align results items with process
  items? Currently 1.1 is aligned with 7.6 instead
  of being aligned with 7.1.

Question 3

• Without comparisons, how do you interpret levels?
  Without a point of reference, how can an Examiner
  understand a level?

55
Question 4

• How are key themes assessed (from applicants
  perspective)?

Question 5

• How do we evaluate applicants that are part of a
  larger system? For example an applicant might
  say that they use a process established by the
  system for which they are a part.

56
Question 6

• Why dont we ask applicants to define/describe
  their work system(s), e.g., in 6.1.a.1, What
  is/are your work system(s)?

Question7

• Clarify the difference between work systems and
  work processes. There appears to be overlap and
  the category is labeled process management.

57
Core Competencies

58
Information resources

• WSQA comment guideline handout
• Your back-up, team lead, PEPD mentor
• Reference materials in
• Baldrige web site http//baldrige.nist.gov/Examine
  r_Resources.htm
• WSQA Website
• http//www.wsqa.net/extraining.php

59
What weve covered today

• WSQA overview
• Baldrige criteria changes
• WSQA ethics review
• WSQA Lite versus Full process
• WSQA process review
• Questions from National Training

60

• THANK YOU!!
• Your support and participation as Examiners helps
  us all by helping WSQA fulfill its mission!