Examiner Training - PowerPoint PPT Presentation

1 / 62
About This Presentation
Title:

Examiner Training

Description:

... great deal of the information examiners need to know has already been covered in ... Only comment on the aspects that are particularly strong ... – PowerPoint PPT presentation

Number of Views:108
Avg rating:3.0/5.0
Slides: 63
Provided by: vacol
Category:

less

Transcript and Presenter's Notes

Title: Examiner Training


1
Examiner Training
  • Process comments
  • Results comments
  • Scoring

2
Agenda
  • Classroom training (Examination focus) Day 3
  • Key Factors
  • Process Items
  • Strength Comments
  • Opportunity for Improvement Comments
  • Results Items
  • Strength Comments
  • Opportunity for Improvement Comments

3
Agenda
  • Classroom Training (Examination focus) Day 4
  • Scoring process for Process and Results Items
  • Overview of what to expect as a member of the
    Board of Examiners
  • Graduation

4
Agenda
  • Classroom Training (Examination focus) Day 4
  • Scoring process for Process and Results Items
  • Overview of what to expect as a member of the
    Board of Examiners
  • Graduation

5
General information
  • There were two trophy winners in 2009. The
    winners were Bay Pines VA Healthcare System and
    the Louisville VA Medical Center. The Bay Pines
    2009 application will be used as the case study
    for this section of the training.
  • Sections of the Bay Pines VAMC application to be
    used in training include
  • Organizational Profile
  • Responses to Items 1.2, 2.1, and 7.6

6
What have we already learned?
  • A great deal of the information examiners need to
    know has already been covered in this class.
    This includes
  • Understanding Approach, Deployment, Learning, and
    Integration
  • Understanding Levels, Trends, Comparisons, and
    Integration
  • Developing and using a Key Factors Worksheet

7
We also learned
  • The following are NOT considered when assessing
    an application
  • Organization of the application. As an examiner,
    if you can find the information needed then you
    give the applicant credit
  • Typos
  • Incorrect figure numbers
  • Poor quality binding

8
General Examination Process
  • Review application to determine if you have a
    conflict of interest
  • Conflicts include
  • You examined this applicant within the past three
    years
  • You worked for this applicant within the past
    three years
  • Your close friend or relative either works at the
    applicant site or is a customer of the site.
  • This includes the recent past. For example, my
    dad was a patient at X. He past away two
    months ago.
  • NOT a conflict
  • You examined the applicant five years ago
  • You worked for the applicant 8 years ago
  • You participated in a SOARS or OAI site visit to
    the applicant last year

9
General Examination Process
  • Read through the entire application (a quick
    read)
  • Develop a Key Factors Worksheet
  • Associate Key Factors with a Criteria Item
  • Examine the Item responses and compare to the
    Criteria
  • Take notes in the application book
  • Draft Strength and OFI comments
  • Repeat steps 3 through 5 for all Items
  • Score all Items

10
Developing a Key Factors Worksheet
  • Read Organizational Profile questions in the
    Criteria book
  • Review Organizational Profile responses in the
    application and underline key facts
  • Draft a Key Factors Worksheet by recording the
    key facts about the applicant
  • This Key Factors Worksheet is a key tool in
    examining an application

11
Developing a Key Factors Worksheet
  • Exercise 20 minutes
  • Review Organizational Profile responses in the
    application
  • Underline key facts
  • We will use this marked-up Organizational Profile
    as our Key Factors Worksheet for purposes of the
    training class

12
Associate Key Factors with an Applicants Item
Responses
  • Exercise 10 minutes
  • Read the Overall Item Requirements for 1.2
  • Review the applicants Organizational Profile
  • Select the 4 6 underlined facts that you
    believe are relevant when evaluating the
    applicants responses to this Item
  • Record these facts at the topic of a piece of
    paper

13
Taking Notes in an Application
  • Exercise 20 minutes
  • Evaluate 1.2b(12) and 1.2c(12)
  • Read the applicants responses to Item 1.2
  • For each Multiple Requirement within the Criteria
  • Read the criteria question
  • Read the applicants response
  • Write A or A-, D or D-, L, or L- in the margin
    next to the applicants response to indicate if
    the scoring factor was positively addressed or not

14
Process Strength Comments
  • Strength comments
  • Are used when the applicant provides evidence of
    ADLI in reference to the criteria
  • Strength comments contain two parts
  • An initial phase or sentence linking the strength
    to the criteria
  • A final phrase or sentence providing evidence
    (examples) from the application to support the
    strength

15
Process Strength Comments, Part One
  • Verb(s) and specific Requirements from the
    Criteria
  • review organizational performance
  • Communicates, listens, and learns
  • Makes needed data and information available
  • Subject from the Criteria or application
  • The senior leaders
  • The organization
  • The school
  • The medical center
  • The cemetery
  • The applicant

16
Process Strength Comments, Part One
  • EXAMPLE
  • 1.1.a(1) Senior leaders set and deploy the
    organizations vision and values through the
    leadership system to the workforce...
  • Note When you pull from the Criteria to write
    your comment, you must be accurate. The Criteria
    says to the workforce, to key suppliers and
    partners, to patients and to other customers.
    In this case there is the link to the Criteria
    and that is what is evidenced in the application.

17
Process Strength Comments, Part One
  • EXAMPLE
  • 1.1.a(1) Senior leaders set and deploy the
    organizations vision and values through the
    leadership system to the workforce...
  • Note
  • Address Item 1.1, Area a, multiple
    requirement (1)
  • Question Which aspects are addressed in this
    comment? A, D, L, and/or I

18
Process Strength Comments, Part Two
  • Example(s) from the application
  • Provide the activities or processes that led you
    to the conclusion that this was a strength the
    facts, the WHAT, the HOW
  • Transition
  • By/through/using
  • This is demonstrated by

19
Process Strength Comments, Part Two
  • by using an annual Strategic Planning process
    and numerous communication methods, such as news
    letters, meetings, and stakeholder sessions to
    communicate to all faculty, staff, students, and
    key stakeholders and partners. The leadership
    system involves many participants and
    decision-making levels.
  • Note This information comes from the
    application.

20
Process Strength Comment - Tips
  • Your comment can be about one or more of the four
    aspects (ADLI)
  • Only comment on the aspects that are particularly
    strong
  • Support your findings with examples from the
    application
  • Review the scoring guidelines does your comment
    align with the performance level that you feel is
    representative of the applicant?

21
Process Strength Comments
  • Exercise 30 minutes
  • Review your 1.2b(12) notes
  • Draft two Strength comments
  • For each comment
  • Underline the link to the criteria
  • Circle the examples from the application
  • Identify a speaker for the group

22
Process Opportunities for Improvement (OFI)
Comments
  • The second part of an Item level examination is
    the identification and preparation of OFIs
  • An OFI identifies an area (usually one factor) in
    the applicants response that in some way does
    not fully respond to a criteria question and may
    be of value to the applicant in its efforts to
    achieve its business objectives
  • How does an examiner determine if a possible OFI
    is of value to the applicant?

23
Process Opportunities for Improvement Comments
  • OFIs contain two parts
  • A statement containing the Criteria and examples
    from the application
  • A statement describing the possible impact on the
    organization

24
Process Opportunities for Improvement Comments
Part One
  • Subject from the Criteria or Application
  • The senior leaders
  • The organizations
  • The medical center
  • The MSN
  • The regional office
  • Introductory Statement
  • It is not clear how/what
  • It is unclear how/what
  • It is not evident how/what
  • It is not apparent how/what

25
Process Opportunities for Improvement Comments
Part One
  • Verb(s) and specific requirements from the
    Criteria
  • Review organizational performance
  • Communicates, listens, and learns
  • Makes needed data and information available
  • Complaint management process

26
Process Opportunities for Improvement Comments
Part Two
  • The second part of an Opportunity for Improvement
    comment is a statement describing the possible
    impact on the organization. This is often
    referred to as a so what statement
  • Why dont we use so what statements in Strength
    comments?
  • Why do we use so what statements in an
    Opportunity for Improvement comments?

27
Process Opportunities for Improvement Comments
Part Two
  • The so what statement describes the benefit or
    consequence associated with the Opportunity for
    Improvement.
  • If you know the part of the criteria that is not
    addressed in the applicants response you could
    look up a standard OFI response in a Baldrige
    reference book
  • For example, an applicant does not gather
    information from customers of competitors. The
    so what might be,
  • Without information from customers of competitors
    leaders may be hindered in their ability to win
    over these customers and gain market share.

28
Process Opportunities for Improvement Comments
Part Two
  • The applicant puts a great deal of time and
    effort into their application
  • Examiners want to provide value-added information
    to the applicant
  • Value-added OFIs are linked to areas that have
    been identified as important to the applicant
  • So what statements need to be specific enough
    and important enough to motivate applicant
    leadership to act
  • A generic OFI from a reference book may be a
    start but it is not specific enough to provide
    the level of importance needed to facilitate
    action
  • ALSO, MOST APPLICANTS ARE SMART ENOUGH TO
    IDENTIFY GENERAL OFI COMMENTS

29
Process Opportunities for Improvement Comments
Part Two
  • So, where does the information to support a good
    so what statement come from?
  • Answer The Organizational Profile - the Key
    Factors Worksheet

30
Process Opportunities for Improvement Comments
Part Two
  • Revised OFI comment
  • Without information from customers of competitors
    leaders may be hindered in their ability to
    identify new products and services that may be
    critical keeping current customers that have a
    choice in health care providers.

31
Examining an Application
  • Exercise 20 minutes
  • Review the criteria, the applicants response,
    and your notes for Item 1.2.c(12)
  • Identify one potential OFI
  • Draft an OFI comment
  • Use the standard two-part comment structure
  • Underline the first part of the comment
  • Circle the so what portion of the OFI
  • Identify a speaker for the group

32
Process Examination
  • Exercise 30 minutes
  • Read Criteria Overall Requirements for Item 2.1
  • Select 4 6 Key Factors
  • Take notes on 2.1a(1)
  • Draft 1 Strength comment
  • Draft one OFI comment

33
Results Examination
  • Examining the Results Items
  • Similar to evaluating process Items except the
    scoring factors change. The examination scoring
    factors are
  • Levels (Le)
  • Trends (T)
  • Comparisons (C)
  • Integration (I)

34
Example showing Level
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
35
Example showing Levels with Segmentation
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
36
Example showing Levels, Segments, and Trends
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
37
Example showing Levels, Segments, Trends, and
Comparisons
Benchmark
Satisfied or Extremely Satisfied
Competitor
Industry Average
Figure 7.1-1 Customer Satisfaction with Product
Quality
38
Category 7Example of Strength Comment
  • Class Exercise
  • What scoring aspects are addressed in this
    strength comment?
  • 7.6a(3) Over the last three years, the
    applicant has increased the percentage of
    external board members from 25 to 60 (Figure
    6.7-1) and has appointed an external director as
    the head of its audit committee. These results
    are related to the applicants strategy of
    achieving greater independence in oversight of
    governance and financial audits.

39
Results Strength Comments
  • Results Strength comments are similar in
    structure to Process Strength comments.
  • There are two parts to a Results Strength
    comment.
  • The first part references the scoring aspect that
    is the focus of the comment (Levels, trends,
    Comparisons, and/or Integration).
  • The second part of the comment provides examples
    (evidence) from the application to support the
    finding.

40
Category 7Examination of Results
  • The keys to successful examination of Items in
    Category 7 are
  • Identifying Expected Results and Key factors
  • Assessing performance levels in relation to
    scoring aspects
  • What results do you expect to find in Category 7?
  • For results shown, assess in terms of (Le, T, C,
    and I)

41
Category 7Results Comments
  • Comment Structure
  • Elements subject (applicant or result),
    timeframe (in, to/from), numbers (be specific),
    figure reference number (always cited), and so
    what statement (if applicable)
  • All key areas of the Item responses should be
    reflected in the Item feedback comments
  • Scoring Aspects all scoring aspects (Le, T, C,
    I) should be addressed in comments

42
Category 7Example of OFI Comment
  • Exercise
  • What scoring aspects are addressed in this OFI
    comment?
  • Although overall satisfaction levels are shown,
    no segmented results are provided. The applicant
    identified inpatient, elderly women, and
    long-term care as key customer segments. Without
    results for these segments it may be difficult
    for leaders to assess progress in meeting the
    unique needs identified as important to
    satisfying these customers and meeting business
    objectives.

43
Category 7Examination of Results
  • The keys to successful examination of Items in
    Category 7 are
  • Identifying Expected Results and Key factors
  • Assessing performance levels in relation to
    scoring aspects.
  • What results do you expect to find in Category 7?
  • For results shown, assess in terms of (Le, T, C,
    I, and Gaps)

44
Category 7Example of Strength Comment
  • Exercise
  • What scoring aspects are addressed in this
    strength comment?
  • Overall satisfaction levels are consistently
    higher than VHA average in each of the three key
    customer requirements (access, courtesy, and
    quality of care) as shown in Figure 7.2-3. The
    Overall Customer Satisfaction Access levels are
    in the mid-90s for the past three years,
    consistently above VHA average (see Figure
    7.2-4). The Overall Customer Satisfaction
    Courtesy levels show an increase from 80 in 2003
    to 88 in 2005 (see Figure 7.2-5). The Overall
    Customer Satisfaction Quality of Care levels
    are sustained at high levels of satisfaction, 97
    each of the past three years (see Figure 7.2-6).
    This places the applicant in the top 10 VHA
    facilities each year for quality of care.

45
Category 7Example of OFI Comment
  • Exercise
  • What scoring aspects are addressed in this OFI
    comment?
  • 7.1a While the applicant shows overall
    customer satisfaction with product quality for
    2002, no trend data are presented to allow it to
    track the rate of improvement (Figure 7.1-1). In
    addition, the data are not segmented by key
    customer group (i.e., industrial or commercial).
    Thus, it is not clear how the applicant
    identifies variation in its customer satisfaction
    over time or from sector to sector to target
    opportunities for improvement.

46
Category 7Example of OFI Comment
  • Exercise
  • What scoring aspects are addressed in this OFI
    comment?
  • Although overall satisfaction levels are shown,
    no segmented results are provided. The applicant
    identified inpatient, elderly women, and
    long-term care as key customer segments. Without
    results for these segments it may be difficult
    for leaders to assess progress in meeting the
    unique needs identified as important to
    satisfying these customers and meeting business
    objectives.

47
Results Comments
  • Exercise 20 minutes
  • Review Overall Requirements for Item 7.6
  • Identify Expected Results
  • Identify key Factors
  • Evaluate information provided using Le, T, C, and
    I
  • Draft one Strength comment
  • Draft one OFI comment

48
Scoring Items
  • Scoring can easy or difficult
  • If you have well-written comments that cover the
    Item criteria and areas most important to the
    applicant, then scoring is easy.
  • If comments are general focused on a narrow
    portion of the Item Criteria concentrated on one
    or two examination aspects and not aligned with
    areas of importance to the applicant then scoring
    is difficult.

49
Scoring Process
  • Individual
  • Review all comments (Strengths and Opportunities
    for Improvement) associated with an Item
  • Turn to the appropriate set of scoring guidelines
    in the Baldrige criteria book or use scoring
    sheets
  • Begin with Approach (A) at the 50 65 score
    range
  • Read the score range description

50
Scoring Process
  • Individual
  • Is this description consistent with comments in
    your analysis of the Item?
  • If yes, read the range above and below to ensure
    you are comfortable with the range selected.
    Indicate if the applicant is in the high, middle,
    or low end of the selected range.
  • If no, move up or down until you read a range
    that is consistent with your Item comments.
    Then, read the range above and below to ensure
    you are comfortable with the range selected.
    Indicate if applicant is in the high, middle, or
    low end of the selected range.

51
Scoring Process
  • Individual
  • Repeat this process for each of the scoring
    aspects (A, D, L, I for Process Items and Le, T,
    C, I or Results Items).
  • Once score ranges have been selected for all four
    scoring aspects, select an overall score range
    that you believe represents the applicants
    demonstrated performance in this area.
  • Review comments and scores.
  • Select a percent score (score in increments of
    5). For example 30, 40, or 45.

52
Scoring Process
  • Individual
  • Repeat this process for each Item
  • Team
  • Place Item level scores in a table displaying the
    scores of everyone on the team.
  • Calculate/estimate the average score for the
    Item.
  • If individual scores are within plus or minus 10
    of the average, average all scores and move on to
    the next Item.

53
Scoring Process
  • If scores vary by more than 10 from the average
    score the team needs to discuss, understand, and
    reach consensus
  • Have the person with high score present facts
    supporting the higher score (linked to the four
    scoring aspects)
  • Have the person with the low score present facts
    supporting the lower score (linked to the four
    scoring aspects)
  • The entire team listens, considers, and rescores

54
Scoring Process
  • Things to consider
  • 10 averaged with 4 other people on the team
    amounts to about a 2 change in score.
  • 2 of an Item is about 1.5 points.
  • There are a total of 1000 possible points.
  • How long do you want to spend trying to reach
    consensus on 2 points?
  • Carey specific procedure
  • If individual scores are more than 15 apart for
    any individual Item, that Item will be flagged
    for the judge. A detailed review of the
    applicant responses, team comments, and score
    will be conducted and the judge will determine
    what changes are needed.

55
Scoring
  • Exercise Part I 15 minutes
  • Case study comments for Item 1.1
  • Review all comments for Item 1.1
  • Individually score using process discussed
  • Share individual scores with people at your table
  • Record individual scores
  • Discuss
  • Rescore
  • Record consensus score

56
Scoring
  • Exercise Part II 15 minutes
  • Case study comments for Item 7.6
  • Review all comments for Item 7.6
  • Individually score using process discussed
  • Share individual scores with people at your table
  • Record individual scores
  • Discuss
  • Rescore
  • Record consensus score

57
Carey Examination Process Flow
  • Examiners
  • Selected and trained
  • Assigned to teams (4-7 on a team)
  • Receive application, conflict of interest form,
    reporting instructions in the mail
  • 2 3 days of independent work prior to team work
    activities face-to-face
  • Read application
  • Key Factors Worksheet
  • Notes in Application
  • Draft comments for assigned Items

58
Carey Examination Process Flow
  • 3.5 days in Washington area for face-to-face
    meeting
  • Examine application - consensus on comments and
    scores
  • Produce a draft feedback report
  • Report to Carey process oversight group (program
    manager and representative from each of the three
    administrations)
  • Oversight group reviews all work and decides
  • Which applications are recommended for site visit
  • Which applications are not recommended for site
    visit
  • Which teams may have had an issue during the week
  • Which teams did an unacceptable job and require
    another examination
  • EVERY MEMBER IS EMPOWERED TO FLAG AN APPLICATION
    FOR ANOTHER REVIEW IF NEEDED

59
Carey Examination Process Flow
  • All applications and team analyses sent to an
    objective, external, Baldrige expert, serving as
    a Carey judge. Judges review the feedback report
    and scores. Judges then calibrate scores and
    comments to ensure a fair and objective
    assessment of each applicant
  • Site visit conducted at facilities that have the
    potential of being recommended for the trophy
    award
  • All applicants scoring over 330 are evaluated to
    ensure that there are no issues that might
    otherwise exclude them from being recognized by
    the Secretary. This could eliminate a fair
    number of potential award recipients
  • Feedback reports prepared and sent to applicant
    organizations

60
Examining Process Flow
  • A list of those remaining applicants with scores
    over 330 is sent to the Secretary with
    performance (score information), a summary of
    applicant best practices, and general
    recommendations (330 430 Achievement, 431 and
    over Excellence or Trophy).
  • Recommendations are considered, then the
    Secretary selects Achievement, Excellence, and
    Trophy winners
  • Symposium and Awards Ceremony in fall

61
Post test
  • Please take 5 - 10 minutes to complete the post
    test.
  • Review and correct in class

62
THANK YOU
  • Have a safe trip home
Write a Comment
User Comments (0)
About PowerShow.com