Performance Management Presentation Provide Animal Research Services

1
Performance Management PresentationProvide
Animal Research Services

• Team Members
• Dr. Charmaine Foltz, Dr. James Crowell, Dr. John
  Bacher, Dr. Michael Eckhaus, Dr. Myrna Mandel,
  Fran Benedetti, Karen Sillers, Thom Spencer, Dr.
  William Rall, Dr. Carl Hansen, Pat Schmidt, Dr.
  Tanya Burkholder
• ORS
• National Institutes of Health
• 7 January 2004

2
Table of Contents

• Main Presentation
• Performance Management Plan
• Relationship Among Performance Objectives
• Customer Perspective
• C1a Number of customer requests by type and
  discrete service
• C1b Number of customer requests addressed by
  type and discrete service
• C2a ORS Customer Scorecard ratings for
  phenotyping
• Internal Business Process Perspective
• IB1a For animal diagnostic services, number
  of health submissions (i.e., histopathologies
  performed) by month
• IB1b For animal health surveillance, number of
  submissions (i.e., comprehensive full panels
  performed) by month

3
Table of Contents (cont.)
Main Presentation Internal Business Process
Perspective IB2a IB2a For animal diagnostic
services, average turnaround time for
histopathologies performed by month IB2b
For animal health surveillance, average
turnaround time for comprehensive full panels
performed by month IB2c For animal health
surveillance, average turnaround time for routine
and rush PCRs performed by month Learning
and Growth Perspective LG1a Percentage of
staff with desired credentials
4
Table of Contents (cont.)
Main Presentation Financial Perspective F1a
Animal transportation unit cost F1c Animal
procurement unit cost F2a Average number of
transportation runs per person by month F2b
Average number of procurement orders processed
per person by month Conclusions Appendix
5
Performance Management Plan
Performance Management Plan (PMP)
Division Approval/Date July 2003
Associate Director Approval/Date July 2003
Service Group
Provide animal research services
Discrete Services
DS1 Conduct animal diagnostic services
DS2 Conduct animal health surveillance
DS3 Perform animal model preservation and
characterization
DS4 Provide clinical animal research services
Value Proposition
Animal Research Services facilitates the NIH
intramural research effort by providing
cost-effective, reliable, and specialized
research support services
delivered by an experienced, responsive, and
professional staff with a reputation for high
quality work.
Service Strategy
Operational Excellence
Growth
Customer Intimacy
Sustain
Product Leadership
Harvest
Strategy Description
To facilitate the NIH intramural research effort,
we conduct diagnostic evaluations, monitor animal
health status, perform surgery, provide intensive
care, operate a pharmacy, and preserve and
evaluate animal models for future research use.
We will focus on streamlining selected internal
business
processes to improve the speed, accuracy, and
cost-effectiveness of our service delivery to our
customers.
Team Leader
Dr. James Crowell and Dr. Charmaine Foltz
Team Members
Dr. John Bacher, Dr. Michael Eckhaus, Dr. Myrna
Mandel, Fran Benedetti, Karen Sillers, Thom
Spencer, Dr. William Rall, Dr. Carl Hansen, Pat
Schmidt, Dr. Tanya Burkholder
Date 12/17/03
6
Relationship Among Performance Objectives
--- relationship between objectives in different
perspectives ___ relationship between objectives
within a perspective most important objectives
7
Customer Perspective
8
Customer Perspective
9
C1a Customer Requests by Type and Discrete
Service (as of FY04Q1)

• Developed log of customer requests
• Clinical Animal Research Services chosen to pilot
  log
• Period of time covered July FY03 through
  November FY04
• Categories of requests based on sample data
• Log will be revised and categories customized for
  each discrete service

10
C1b Customer Requests Addressed (as of FY04Q1)

• Customer log will also track customer requests
  addressed

11
Customer Perspective (cont.)
12
Customer Scorecard Methodology

• In FY03, modified ORS Customer Scorecard for
  phenotyping service
• E-mailed survey in Quarter 1 FY04 to all
  customers using this service in FY03 (i.e., 33
  customers)
• Each fiscal year, a different service(s) will be
  chosen for measurement

13
C2a ORS Customer Scorecard Ratings for
Phenotyping (as of FY04 Q1)
N 17
Satisfaction Mean Ratings
14
In FY04 we will charge for this service.
Commercially, phenotyping is available at a cost
of about 14,000. If we were to offer this
service to you at a cost of 7,500 how likely is
it that you would use our service? (1Not at all
likely 10Extremely likely)
C2a ORS Customer Scorecard Ratings for
Phenotyping (as of FY04 Q1) (cont.)
N 17 Mean 5.2 Median 5-1/2
15
What was done well?
C2a ORS Customer Scorecard Ratings for
Phenotyping (as of FY04 Q1) (cont.)

• Digitizing the image of the sample and emailing
• Preparation of slides from tissues and
  pathological analysis
• All was fine
• The service is fantastic and very much needed
• Thoroughness and consultation with pathologist
  after evaluation of mice .
• The general analysis was thorough and Dr. Miller
  was quick to respond to any questions/concerns
  that I raised regarding the phenotyping project.
  While I utilized the free service in FY03, it was
  still useful and has generated many leads into
  understanding the mechanism behind the phenotype
  of the mice. The quality of the slides is good
  yet the blood smears could be a better. However,
  using the DiffSpin apparatus will likely
  alleviate these problems.
• Thorough histological analyses
• Histological analysis of primary phase
• The willingness to deal with unique problems
  (i.e. removing superior cervical ganglia from
  mice) and to try to work with us on anything
  unusual.
• Whole body survey and material handling.

16
What needs to be improved?
C2a ORS Customer Scorecard Ratings for
Phenotyping (as of FY04 Q1) (cont.)

• Speed of analysis
• All was fine
• Great as is
• It would be useful to supply a key for the slides
  showing the location of tissues.
• Everything was fine.
• Seemed to take a while to get the first analysis
  results(for the MT1 knockout line) but the other
  results were done more rapidly (for MT4 and MTB
  knockout lines).
• Nothing. It is great.
• Digital picture of tissues would be a plus.

17
Other Comments
C2a ORS Customer Scorecard Ratings for
Phenotyping (as of FY04 Q1) (cont.)

• Although our genetically-modified animals did not
  show any consistent abnormalities, this service
  was very useful in helping to characterize the
  phenotype of the mice.
• Number of projects will decrease due to
  prioritization
• Have used the service three times. In one case a
  very interesting and important phenotype was
  found. In the other cases no obvious phenotype
  was noted but this was still valuable
  information. All personnel involved were
  professional and a pleasure to work with.
• I think that the price of the service should be
  based on the number of mice analyzed. It might
  not be necessary in all cases to look at all
  genotypes (wt, heterozygotes, and knockouts) so
  it might be more feasible for the investigator if
  the cost were calculated from the number of mice.
  
• 7,500 is a bit much.
• We checked very few animals therefore the
  phenotyping was not sufficiently useful.
• If I really needed it to be done I would do it
  here. But 7500 is a lot of money..

18
Customer Perspective Summary

• The data suggests there is a need to
  systematically collect and analyze customer
  requests in order to determine whether we are
  providing the right mix of services
• Based on FY02 analysis, the pathology phenotyping
  service was initiated in May, 2002
• Goal was to phenotype approximately 30 animal
  models in FY03
• In FY02 15 models were phenotyped
• In FY03, 33 models were phenotyped which exceeded
  our goal

19
Customer Perspective Summary (cont.)

• The response rate for the Phenotyping survey is
  51 of all FY03 customers for this service.
• Overall, customers indicated they were highly
  satisfied on all satisfaction dimensions.
• Mean ratings ranged from 8.5 to 9.0 on a 10-point
  scale.
• However, customers are not likely to purchase the
  service.
• Only one respondent indicated that he/she is
  highly likely to purchase the service at the
  suggested price.
• Seven respondents indicated a moderate likelihood
  to purchase (I.e, ratings of 4 7 on a 10-point
  scale)

20
Customer Perspective Planned Actions

• Revise and implement customer log of requests and
  commendations/complaints
• Continue to evaluate requests for new services
  and implement as feasible
• Address complaints and take appropriate actions
  to improve service
• Survey selected service customers on a rotating

21
Customer Perspective Planned Actions (cont.)

• Present Phenotyping survey results to Shared
  Resources Subcommittee (Dec. 03) and Scientific
  Directors (Jan. 04)

22
Internal Business Process Perspective
23
Internal Business Process Perspective
24
IB1a Number of Histopathologies Performed by
Month (as of 30 Sept)
Average 159 Range 75 - 250
Note Control chart (IB1a1) shown in Appendix
25
IB1b Number of Comprehensive Full Panels
Performed by Month (as of 30 Sept)
Average 474 Range 279 - 736
Note Control chart (IB1b1) shown in Appendix
26
Internal Business Process Perspective
27
IB2a Average Turnaround Time for
Histopathologies Performed by Month (as of FY04Q1)
Average 30 Range 20 46
Note Control charts (IB2a1) shown in Appendix
28
IB2b Average Turnaround Time for Comprehensive
Full Panels Performed by Month (as of 30 Sept)
Average 10.5 Range 7.75 15.28
Note Control charts (IB2b1) shown in Appendix
29
IB2c Average Turnaround Time for Routine and
Rush PCRs Performed by Month (as of 30 Sept)
Rush Average 3 days Range 1 4
days Routine Average 4 days Range 2 5
days
Note Control charts (IB2c1 and IB2c2) shown in
Appendix
30
Internal Business Process PerspectiveSummary

• Animal Diagnostic Services
• Data analyzed for FY02, FY03, and Oct. of FY04
• An average of 159 histopathologies are performed
  each month
• Average turnaround time is 30 days for entire
  data set. Control chart analysis revealed a
  significant difference in turnaround times
  between FY02 and FY03/Oct. FY04.
• FY02 average turnaround time is 35 days
• FY03/Oct. FY04 average turnaround time is 25 days
• Difference in turnaround time due to staff
  changes.
• It is expected that average turnaround time will
  remain at current level of 25 days.

31
Internal Business Process PerspectiveSummary
(cont.)

• Animal Health Surveillance Comprehensive Full
  Panels
• Data analyzed for FY03
• An average of 474 comprehensive full panels are
  performed each month
• Average turnaround time is 10 days
• Control chart analysis revealed one special cause
  signal (i.e., 15 days turnaround time in Sep.)
  but no explanation is readily available
• The data seem to suggest a larger number of
  submissions in the months which end fiscal
  quarters (i.e., December, March, June, September)

32
Internal Business Process PerspectiveSummary
(cont.)

• Animal Health Surveillance Routine and Rush
  PCRs
• Sample data analyzed for FY03
• Routine PCRs 58 of sample
• Rush PCRs 42 of sample
• Average Turnaround Times
• Routine PCRs 4 days
• Rush PCRs 3 days
• Control chart analysis revealed a special cause
  signal in the routine PCR sample but no
  explanation is readily available

33
Internal Business Process PerspectivePlanned
Actions

• Animal Diagnostic Services
• An average of 159 histopathologies are performed
  each month
• The process is predictable with a normal range of
  about 75 to 250 accessions each month
• Average turnaround time beginning with FY03 is 25
  days with a range between 20 to 44 days
• This time is acceptable, but could be improved.
  Need to investigate ways to further improve
  turnaround time.
• Data from latest reporting periods suggest a
  permanent improvement has already been achieved.
• We will continue to gather data and monitor.

34
Internal Business Process PerspectivePlanned
Actions (cont.)

• Animal Health Surveillance Comprehensive Full
  Panels
• An average of 474 comprehensive full panels are
  performed each month
• The process is predictable but has a large amount
  of natural variation with a range between 279 and
  736 submissions each month
• Average turnaround time beginning with FY03 is
  10.5 days with a range between 7.75 to 15.28 days
• This time is acceptable in most cases - goal is
  12 days or less.
• Need to identify and control for factors causing
  delays. We will continue to gather data and
  monitor.

35
Internal Business Process PerspectivePlanned
Actions (cont.)

• Animal Health Surveillance Routine and Rush
  PCRs
• Average turnaround time for Routine PCRs is 4
  days with a range between 2 to 5 days
• Average turnaround time for Rush PCRs is 3 days
  with a range between 1 to 4 days
• Due to the large percentage of rush jobs and the
  relatively small difference in turnaround time
  between rush and routine jobs, we will be adding
  personnel resources in this area.
• We will continue to gather data and evaluate
  effects of increased resources.

36
Learning and Growth Perspective
37
Learning and Growth Perspective
FY 03
Objective
Measure
FY04 Target
FY05 Target
Initiative
Owner
Target
LG1 Maintain and
LG1a Percentage of staff with
Baseline
TBD
TBD
Identify improvements
Foltz
enhance competencies
desired credentials
and implement
for the future DVR
38
LG1a Percentage of Staff with Desired
Credentials (as of FY04 Q1)
39
Learning and Growth PerspectiveSummary

• Most of the Animal Research Services Group have
  the desired credentials
• The credentials are important to ensure technical
  expertise
• Staff members contributed to 33 publications in
  FY03 either as primary (10) or secondary (23)
  authors
• Staff members conducted 25 presentations outside
  NIH (20 local, 5 national)
• Staff members conducted 19 presentations within
  NIH

40
Learning and Growth PerspectivePlanned Actions

• It is important to maintain a high level of
  expertise within the Animal Research Services
  Group
• In FY04, we plan to support board training for
  all veterinarians.
• Weve done an outstanding job in FY03 of
  demonstrating our capabilities to our
  stakeholders as shown by the considerable number
  of publications and presentations
• In FY04 we plan to continue to use the
  Professional Activities log piloted in FY03

41
Financial Perspective
42
Financial Perspective
43
F1a Unit Cost of Animal Diagnostic Services (as
of 30 Sep)

Note Unit cost increased 4
44
F1b Unit Cost of Health Surveillance Services
(as of 30 Sep)
Note Unit cost decreased 3.6
45
F1c Unit Cost of Animal Model Preservation and
Characterization Services (as of 30 Sep)
Note Unit cost increased 1.2
46
F1d Unit Cost of Clinical Animal Research
Services (as of 30 Sep)

Note Unit cost increased 3
47
Financial PerspectiveSummary

• Animal diagnostic services unit cost
  (submissions/budget)
• Unit cost has increased by 4.0 which is most
  likely due to normal inflationary factors.
• Animal health surveillance unit cost
  (submissions/budget)
• Unit cost has decreased by 3.6 which is within
  the expected variation for this service.
• Animal model preservation and characterization
  services unit cost (models/budget)
• Unit cost has remained essentially the same

48
Financial PerspectiveSummary (cont.)

• Clinical animal research services unit cost has
  remained essentially unchanged.
• Surgery service capacity has increased by 4
  which is within the range of normal variation
• Billing Hours for the ICU has increased by 18
  which is most likely due to more accurate record
  keeping.

49
Financial PerspectivePlanned Actions

• Continue to mange services in cost effective
  manner
• Goal is to ensure only modest cost increases
  commensurate with inflationary pressures
• Identify and monitor cost drivers outside DVR
  control that can threaten overall value of our
  services

50
Conclusions
51
Conclusions from PMP

• Phenotyping initiative was highly successful
• SDs involved in resolving financial issues
• Significantly refined PMP process, providing
  better baseline data for use in future years.
• No direct correlation was found between workload
  and turnaround times.
• Diagnostic case completion times were
  significantly lower in FY03 compared to FY02.
• Unit cost increases were generally lt5
• In general, staff is well qualified and
  participating in professional activities.

52
Appendix

• Appendix
• Pages 2 6 of Performance Management Plan
• Customer Perspective
• C2b Number of complaints and commendations by
  category and discrete service
• Internal Business Process Perspective
• IB1a1 Control chart analysis of
  histopathologies performed by month
• IB1b1 Control chart analysis of comprehensive
  full panels performed by month
• IB2a1 Control chart analysis of histopathology
  turnaround time by month
• IB2b1 Control chart analysis of comprehensive
  full panel turnaround time by month
• IB2c1 Control chart analysis of routine PCR
  turnaround time by month
• IB2c2 Control chart analysis of rush PCR
  turnaround time by month

53
Appendix (cont.)

• Appendix
• Learning and Growth Perspective
• LG2a Number of publications, presentations,
  and other professional activities
• LG2b Number of publications including DVR
  investigator by primary and secondary authorship
• LG2c Number of presentations within and
  outside NIH
• Financial Perspective
• F2a Surgery service capacity
• F2b ICU utilization

54
Performance Management Plan (page 2)
55
Performance Management Plan (page 3)
56
Performance Management Plan (page 4)
57
Performance Management Plan (page 5)
58
Performance Management Plan (page 6)
59
Customer Perspective
60
C2b Customer Feedback Categorization (as of
FY04Q1)

• Developed and pilot tested customer log of
  commendations and complaints
• Clinical Animal Research Services chosen to pilot
  log
• Period of time covered July FY03 through
  November FY04

61
Internal Business Process Perspective
62
IB1a1 Number of Histopathologies Performed by
Month (as of 30 Sept)
63
IB1b1 Number of Comprehensive Full Panels
Performed by Month (as of 30 Sept)
64
Internal Business Process Perspective (cont.)
65
IB2a1 Average Turnaround Time for
Histopathologies Performed by Month (as of FY04Q1)
66
IB2b1 Average Turnaround Time for Comprehensive
Full Panels Performed by Month (as of 30 Sept)
67
IB2c1 Average Turnaround Time for Routine PCRs
Performed by Month (as of 30 Sept)
68
IB2c2 Average Turnaround Time for Rush PCRs
Performed by Month (as of 30 Sept)
69
Learning and Growth Perspective
70
LG2a Number of Publications, Presentations, and
Other Professional Activities (as of 30 Sep)

• Developed and pilot tested Professional
  Activities log
• All discrete services participated
• Period of time covered July FY03 through
  November FY04

71
LG2b Number of Publications by Authorship (as
of 30 Sep)

• Developed and pilot tested Professional
  Activities log
• All discrete services participated
• Period of time covered July FY03 through
  November FY04

72
LG2c Number of Presentations Within and Outside
NIH (as of 30 Sep)

• Developed and pilot tested Professional
  Activities log
• All discrete services participated
• Period of time covered July FY03 through
  November FY04

73
Financial Perspective
74
F2a Surgery Service Capacity (as of 30 Sep)
Note Capacity increased 4
75
F2b ICU Utilization (as of 30 Sep)
Note Overall utilization for all activities in
this area was 56 for FY03