Study on

1
Study on Health System Performance
AssessmentEvidence-based District Performance
Ranking

• Sharing Pakistans Experience, Challenges and
  Policy Implications

Dr. Sohail Amjad Annual Meeting CoP-MfDR Colombo,
November 11-14, 2008
2
Background Introduction

• Publication of countries Health System
  Performance Ranking by WHO-Pakistan stood at
  122nd rank
• Emerging pressure on achievement of MDGs and PRSP
  targets
• Six years since devolved district health system
  in Pakistan, which needs performance assessment
• World Banks PRSC-III indicative triggers for
  implementation of reforms and making system
  result oriented
• Health System Performance Assessment/Analysis on
  key health outputs/outcomes
• Assessment of progress on performance of key
  health priority areas
• Improving Monitoring and Evaluation capacities of
  MoH for evidence-based decision-making

3
Objectives

• To assess the service delivery performance of
  devolved district health system in Pakistan based
  on available quality data sources (PSLM, MICS)
• To develop ranking of the districts based on
  health systems performance

4
HSPA Methodology Options

• Analytic Framework for HSPA by WHO (WHR 2000
  2003 Report)
• Use of fixed weight for composite outcome
  indicators-e.g. DALE, QALE etc, without
  addressing country context and socio-economic
  scenario
• Country ranking raised diverse comments and
  concerns over methodology
• NHS Star Rating
• Use for institutions/hospitals rating
• Specific data needs-economic, service delivery,
  incentives, human resource, quality
• CPIA (Country Policy and Institution Assessment)
  by World Bank
• Use 16 criteria grouped under four clusters
• Used as policy scorecard for rating of country
  policies
• Critics do not recommend its use for HSPA

5
Cont HSPA Methodology Options

• Balance Score Cards/Score Cards
• Score cards are used to summarize statistics for
  the different domains of health systems
  performance assessment
• In Afghanistan, 29 indicators and indexes in six
  domains form a score card
• Indexes are based on uniform agreed scores among
  stakeholders
• Effective Coverage and Co-coverage (link)
  performance assessment method used by MoH Mexico
  in collaboration with Harvard
• Used scores for performance assessment of 13
  interventions
• Uniform and actual scores to all indicators to
  develop Composite Coverage Scores
• Co-coverage score for different interventions
  received by same individual (Measles, DPT3, BCG)
  and (ANC, PNC, TT) was used to develop Composite
  Coverage Index
• Effective coverage, co-coverage, and a composite
  coverage index all have potential to be useful
  metrics for health systems performance assessment

6
The Way Forward

• No specific methodology available to assess
  performance of a developing countrys health
  system
• Developed country methodologies are not relevant
• Recommendations of Report of a Technical Meeting,
  (WHO-HMN) Glion, Switzerland, September 2006
• Promote co-coverage for childhood interventions
  for all countries with DHS and other surveys to
  assess trends and performance
• Further test composite coverage index from
  available aggregate data
• Promote innovative measurement studies in
  countries as part of health systems performance
  assessment
• Methodology need to be tailored for Pakistans
  HSPA due to data constraints

7
Proposed Study Design and Methodology
Health System Ranking
Country, Provincial and District Performance
Ranking (Using Band 1-5)
Effective Coverage
Combined Performance (Coverage) Index (CPI)
Cluster Coverage
Performance Index C
Performance Index A
Performance Index B
Co-coverage
Interventions Scored
Interventions Scored
Interventions Scored
Interventions Scored
Interventions Scored
Interventions Scored
Facility-based Utilization
Domains
Out-reach Services
Quality of Services
RH Services
Other
EPI
Individual Interventions
Individual Interventions
Individual Interventions
Individual Interventions
Individual Interventions
Indicators
8
Methodological Framework
S.No Indicators/ Interventions Domains Co-coverage Clusters Performance Index (PI) Combined Performance Index (CPI)
1 age of fully immunized children 12-23 months EPI (Domain 1) Co-Coverage A Performance Index A CPI PI API BPI C
2 age of DPT3 coverage children 12-23 months EPI (Domain 1) Co-Coverage A Performance Index A CPI PI API BPI C
3 age of pre-natal consultations by LHW Outreach RH Services (Domain 2) Co-Coverage B Performance Index B CPI PI API BPI C
4 Contraceptive prevalence rate Outreach RH Services (Domain 2) Co-Coverage B Performance Index B CPI PI API BPI C
5 age maternal TT coverage Facility-based RH Services (Domain 3) Co-Coverage B Performance Index B CPI PI API BPI C
6 age of pre-natal consultation at PHF Facility-based RH Services (Domain 3) Co-Coverage B Performance Index B CPI PI API BPI C
7 age of PNC consultation at PHF Facility-based RH Services (Domain 3) Co-Coverage B Performance Index B CPI PI API BPI C
8 age deliveries conducted at PHF Facility-based RH Services (Domain 3) Co-Coverage B Performance Index B CPI PI API BPI C
9 age of health consultations at PHF Health Consultation (Domain 4) Co-Coverage C Performance Index C CPI PI API BPI C
9
Material Methods

• Listing all health related indicators collected
  in PSLM (2004-05), and MICS (2004)
• Short listing of Indicators (Justification)
• SMART Criteria
• Only those preferred based on recall and record
• Services utilized or provided by public health
  system
• Relevant to MDGs PRSP targets
• Reflection of both National Programs and routine
  services Performance
• Grouping under domains (e.g. EPI, RH services,
  Health Consultations etc) and co-coverage
  clusters
• Each domain will have 2-3 indicators (for
  specific interventions)
• 1-2 domains in each Co-coverage cluster
• Mapping and data entry for each province and
  district

10
Cont Material Methods

• Awarding 1 point or score for each 10 decimal of
  achievement e.g. Measles 76 award 7.6 points or
  scores, ANC 34 earns score of 3.4 on 1-10 scale
• Sum up each Co-coverage cluster scores to
  generate Performance (Coverage) Indexes A, B C
  
• Useful to analyze performance of different
  interventional domains and Co-coverage clusters
• Provide comparison of out-reach and
  facility-based services performance
• Finally sum up PI A, B C to develop Combined
  Performance (Effective Coverage) Index
• Based on CPI-Rank country districts into five
  bands from 1-5 (Very low performance to very high
  performance)
• Final national and provincial district ranking

11
Findings and Results
Performance Ranking Color Codes
Name of District CPI (Band Scale 1-50) Level of Health System Performance Performance Ranking
A CPI 41-50 Very High Performing Districts 1
B CPI 31-40 High Performing Districts 2
X CPI 21-30 Medium Performing Districts 3
Y CPI 11-20 Low Performing Districts 4
Z CPI 1-10 Very Low Performing Districts 5
12
HSP and Districts Ranking in Punjab
High Performing Districts
Very High Performing Districts
District CPI Final Ranking
Lahore 40 6
Khushab 39 7
Sargodha 38 8
Chakwal 37 9
Mianwali 36 10
Bahawalnagar 35 11
Sahiwal 35 12
Layyah 35 13
Gujranwala 34 14
Gujrat 34 15
Bhakar 33 16
T.T. Singh 33 17
Kasur 33 18
Vehari 33 19
Multan 32 20
Hafizabad 31 21
Faisalabad 31 22
Jhang 31 23
Mandi Bahuddin 31 24
Sheikhupura 31 25
District CPI Final Ranking
Rawalpindi 46 1
Jehlum 44 2
Attock 42 3
Narowal 42 4
Sialkot 41 5
Medium Performing Districts
District CPI Final Ranking
Okara 30 26
Lodhran 30 27
Bahawalpur 30 28
R.Y. Khan 30 29
Pakpattan 29 30
Muzaffar Garh 28 31
Khanewal 28 32
D.G. Khan 27 33
Rajanpur 24 34
13
HSP and Districts Ranking in Sindh
Medium Performing Districts
High Performing Districts
District CPI Final Ranking
Sukkur 30 5
Thatta 30 6
Khairpur 29 7
Shikarpur 29 8
Larkana 28 9
Dadu 28 10
Tharparkar 28 11
Badin 27 12
Ghotki 27 13
Nawabshah 22 14
District CPI Final Ranking
Karachi 37 1
Hyderabad 35 2
Nowshero Feroze 34 3
Mir Pur Khas 31 4
Low Performing Districts
District CPI Final Ranking
Sanghar 20 15
Jaccobabad 16 16
14
HSP and Districts Ranking in Balochistan
High Performing Districts
Very High Performing Districts
District CPI Final Ranking
Gwadar 47 1
Kharan 45 2
Awaran 44 3
Changhi 44 4
District CPI Final Ranking
Pashin 37 5
Kalat 36 6
Ketch 35 7
Bolan 35 8
Zhob 34 9
Loralai 34 10
Ziarat 33 11
Panjgur 31 12
Medium Performing Districts
Low Performing Districts
District CPI Final Ranking
Jhal Magsi 30 13
Quetta 29 14
Mastung 29 15
Lasbilla 28 16
Sibbi 26 17
Jafarabad 23 18
District CPI Final Ranking
Khuzdar 19 19
Nasirabad 19 20
Musa Khel 18 21
Qilla Saifullah 16 22
Barkhan 14 23
Qilla Abdulla 14 24
15
HSP and Districts Ranking in NWFP
High Performing Districts
Very High Performing Districts
District CPI Final Ranking
Charsada 43 1
Upper Dir 42 2
Malakand 42 3
Chitral 42 4
Nowshera 41 5
Abbottabad 41 6
District CPI Final Ranking
Swat 40 7
Peshawer 40 8
Haripur 39 9
Mardan 38 10
Lower Dir 38 11
Swabi 38 12
Hangu 35 13
Bannu 34 14
Mansehra 33 15
Kohat 31 16
Medium Performing Districts
District CPI Final Ranking
Bonair 30 17
Karak 29 18
Battagram 28 19
D.I.Khan 27 20
Lakki Marwat 24 21
Shangla 23 22
Tank 21 23
Kohistan 21 24
16
National and Provincial distribution of districts
based on level of HSP
Province VHP Districts Link HP Districts link MP Districts link LP Districts link VLP Districts
Punjab 5 20 9 Nil Nil
Sindh Nil 4 10 2 Nil
NWFP 6 10 8 Nil Nil
Balochistan 4 8 6 6 Nil
Pakistan 15 42 33 8 Nil
17
Percentage distribution of districts based on
level of HSP
Level of HS Performance age of districts by performance age of districts by performance age of districts by performance age of districts by performance age of districts by performance
Level of HS Performance Punjab Sindh NWFP Balochistan Pakistan
VHP 14.7 Nil 25 16.66 15.3
HP 58.82 25 41.66 33.33 42.85
MP 26.47 62.5 33.33 25 33.67
LP Nil 12.5 Nil 25 8.16
VLP Nil Nil Nil Nil Nil
18
Co-coverage (EPI RH) Comparison and Ranking of
Top Ten Performing Districts of Punjab
Top Ten EPI Performing Districts in Punjab Top Ten EPI Performing Districts in Punjab
District Rank
Jehlum 1
Sialkot 2
Khushab 3
Attock 4
Mianwali 5
Gujrat 6
Chakwal 7
Rawalpindi 8
Bahawalnagar 9
Pakpattan 10
Top Ten RH Performing Districts in Punjab Top Ten RH Performing Districts in Punjab
District Rank
Rawalpindi 1
Narowal 2
Jehlum 3
Lahore 4
Sialkot 5
Attock 6
Sargodha 7
Khushab 8
Layyah 9
Kasur 10
CPR Top Ten Punjab
Districts
Lahore
Sialkot
Rawalpindi
Gujranwala
Gujrat
Jehlum
Kasur
Narowal
Faisalabad
Sargodha
19
Co-coverage (EPI RH) Comparison and Ranking of
Top Ten Performing Districts of Sindh
Top Ten EPI Performing Districts in Sindh Top Ten EPI Performing Districts in Sindh
District Rank
Hyderabad 1
Nowshero Feroze 2
Karachi 3
Larkana 4
Shikarpur 5
Dadu 6
Ghotki 7
Badin 8
Sukkur 9
Mir Pur Khas 10
Top Ten RH Performing Districts in Sindh Top Ten RH Performing Districts in Sindh
District Rank
Karachi 1
Nowshero Feroze 2
Hyderabad 3
Mir Pur Khas 4
Thatta 5
Tharparkar 6
Sukkur 7
Khairpur 8
Shikarpur 9
Larkana 10
CPR Top Ten Sindh
Districts
Karachi
Tharparkar
Hyderabad
Sukkur
Nowshero Feroze
Khairpur
Thatta
Mir Pur Khas
Ghotki
Dadu
20
Co-coverage (EPI RH) Comparison and Ranking of
Top Ten Performing Districts of Balochistan
Top Ten EPI Performing Districts in Balochistan Top Ten EPI Performing Districts in Balochistan
District Rank
Gwadar 1
Zhob 2
Ziarat 3
Quetta 4
Kalat 5
Bolan 6
Pashin 7
Awaran 8
Jhal Magsi 9
Kharan 10
Top Ten RH Performing Districts in Balochistan Top Ten RH Performing Districts in Balochistan
District Rank
Changhi 1
Kharan 2
Awaran 3
Gwadar 4
Pashin 5
Panjgur 6
Kalat 7
Ketch 8
Bolan 9
Loralai 10
CPR Top Ten Balochistan
Districts
Lasbilla
Ziarat
Zhob
Changhi
Khuzdar
Gwadar
Jhal Magsi
Kharan
Loralai
Barkhan
21
Co-coverage (EPI RH) Comparison and Ranking of
Top Ten Performing Districts of NWFP
CPR Top Ten NWFP
Districts
Nowshera
Tank
Chitral
Peshawer
Swat
Bannu
Mansehra
Haripur
Abbottabad
Swabi
Top Ten RH Performing Districts in NWFP Top Ten RH Performing Districts in NWFP
District Rank
Nowshera 1
Charsada 2
Malakand 3
Peshawer 4
Upper Dir 5
Abbottabad 6
Mardan 7
Hangu 8
Haripur 9
Swat 10
Top Ten EPI Performing Districts in NWFP Top Ten EPI Performing Districts in NWFP
District Rank
Chitral 1
Abbottabad 2
Swat 3
Charsada 4
Malakand 5
Haripur 6
Upper Dir 7
Swabi 8
Lower Dir 9
Mardan 10
22
Co-coverage (EPI RH) Comparison and Ranking of
Top Ten Performing Districts of Pakistan
Top Ten EPI Performing Districts of Pakistan Top Ten EPI Performing Districts of Pakistan
District Rank
Chitral 1
Jehlum 2
Sialkot 3
Khushab 4
Gwadar 5
Attock 6
Mianwali 7
Gujrat 8
Chakwal 9
Rawalpindi 10
Top Ten RH Performing Districts of Pakistan Top Ten RH Performing Districts of Pakistan
District Rank
Changhi 1
Rawalpindi 2
Kharan 3
Awaran 4
Narowal 5
Nowshera 6
Charsada 7
Malakand 8
Gwadar 9
Jehlum 10
CPR Top Ten Pakistan
Districts
Lahore
Sialkot
Nowshera
Karachi
Rawalpindi
Gujranwala
Gujrat
Tank
Jehlum
Kasur
23
Provincial Comparison of Mean Service Delivery
Performance
Mean Performance Scorecard
Province EPI Performance RH Performance Outreach Services Performance HF (Static Center) Performance
Punjab 17.5 14.7 18.3 12.8
Sindh 14 12 15.4 10.6
Balochistan 12 13.2 12 16
NWFP 15.7 15.5 17 15
24
Conclusion and Recommendations

• A user friendly simple study methodology for a
  developing country setting
• Scope of performance assessment enhanced through
  access to routine and non-routine data sets
• Regular up-dating of information for
  evidence-based resource allocation, if PSLM, or
  MICS reports available
• Multiple aspects of health system performance can
  be looked at a time
• Inter-district and inter-provincial health system
  performance comparison is possible
• Inclusion of indicators for
• Management (human resources, timely procurement
  of supplies and financial management e.g. age of
  budget spent or surrender etc)
• Quality of care, Client satisfaction and
  acceptability of services
• More health-related MDGs PRSP target indicators
  should be included in PSLM surveys
• Gender related service utilization
• Economic quintiles and relationship with service
  utilization

25
Policy Implications

• Coverage as a Measure of Accessibility,
  Affordability and Acceptability
• Coverage as an indirect measurement of
  accessibility and acceptability
• geographical inequalities
• Performance disparities between target
  interventions e.g. EPI vs. Reproductive Health
• Evidence-based Decision-making Resource
  Allocation
• Practical entry point for performance assessment
  is the decision-making cycle, comprising policy
  analysis, goal and target setting, resource
  allocation, work planning, and operational
  implementation through use of information.
• Integration of Health Information Systems
• Regular Tracking of Progress and Achievement
• Opportunity for In-depth Investigation

26
Thanks
27
HSP and Districts Ranking in Pakistan
Very High Performing Districts
District Provincial Distribution Retrn Final Ranking
Gwadar 1
Rawalpindi 2
Kharan 3
Awaran 4
Changhi 5
Jehlum 6
Charsada 7
Attock 8
Upper Dir 9
Narowal 10
Chitral 11
Malakand 12
Abbottabad 13
Sialkot 14
Nowshera Feroze 15
Punjab 5
Sindh None
Balochistan 4
NWFP 6
28
HSP and Districts Ranking in Pakistan
High Performing Districts
District Provincial Distribution Final Ranking
Lahore 16
Peshawer 17
Swat 18
Haripur 19
Khushab 20
Lower Dir 21
Swabi 22
Mardan 23
Sargodha 24
Chakwal 25
Pashin 26
Karachi 27
Kalat 28
Mianwali 29
Bahawalnagar 30
Ketch 31
Hyderabad 32
Sahiwal 33
Punjab 20
Sindh 4
Balochistan 8
NWFP 10
29
HSP and Districts Ranking in Pakistan
Cont High Performing Districts
District Provincial Distribution Retrn Final Ranking
Layyah 34
Bolan 35
Hangu 36
Bannu 37
Gujrat 38
Zhob 39
Nowshero Feroze 40
Gujranwala 41
Loralai 42
Ziarat 43
Bhakar 44
Kasur 45
Vehari 46
Mansehra 47
T.T. Singh 48
Multan 49
Mandi Bahuddin 50
Sheikhupura 51
Hafizabad 52
Kohat 53
Jhang 54
Faisalabad 55
Mir Pur Khas 56
Panjgur 57
Punjab 20
Sindh 4
Balochistan 8
NWFP 10
30
HSP and Districts Ranking in Pakistan
Medium Performing Districts
District Provincial Distribution Final Ranking
Bahawalpur 58
Sukkur 59
Bonair 60
Okara 61
Thatta 62
Lodhran 63
Jhal Magsi 64
R.Y. Khan 65
Shikarpur 66
Quetta 67
Mastung 68
Pakpattan 69
Khairpur 70
Karak 71
Battagram 72
Larkana 73
Tharparkar 74
Lasbilla 75
Punjab 9
Sindh 10
Balochistan 6
NWFP 8
31
HSP and Districts Ranking in Pakistan
Cont Medium Performing Districts
District Provincial Distribution Retrn Final Ranking
Muzaffar Garh 76
Dadu 77
Khanewal 78
D.G. Khan 79
Badin 80
D.I.Khan 81
Ghotki 82
Sibbi 83
Rajanpur 84
Lakki Marwat 85
Jafarabad 86
Shangla 87
Nawabshah 88
Tank 89
Kohistan 90
Punjab 9
Sindh 10
Balochistan 6
NWFP 8
32
HSP and Districts Ranking in Pakistan
Low Performing Districts
District Provincial Distribution Retrn Final Ranking
Sanghar 91
Khuzdar 92
Nasirabad 93
Musa Khel 94
Jaccobabad 95
Qilla Saifullah 96
Barkhan 97
Qilla Abdulla 98
Punjab None
Sindh 2
Balochistan 6
NWFP None
33
Some Important Definitions

• Coverage the percent of people receiving a
  specific intervention among those who need it
• Co-coverage Specific interventions received by
  the same individual
• Effective coverage the probability that an
  individual will receive health gain from an
  intervention if they need it Retrn