Methods to Prioritize Systematic Reviews: The Guide to Community Preventive Services

1
Methods to Prioritize Systematic Reviews The
Guide to Community Preventive Services

• Shawna L. Mercer, M.Sc., Ph.D.
• Director
• The Guide to Community Preventive Services
• National Center for Health Marketing
• Centers for Disease Control and Prevention (CDC)

2
Presentation Structure

• What is the Guide to Community Preventive
  Services (Community Guide)?
• What issues influence the way the Community Guide
  prioritizes its systematic reviews?
• How does the Guide prioritize its work?
• Into topics
• Into interventions within topics that are
  addressed in systematic reviews

3
The Community Guide is

• Systematic reviews of the available evidence
• On effectiveness of population-based public
  health interventions
• Formulated by teams of renowned researchers,
  public health practitioners, representatives of
  health organizations
• Concise, carefully-considered recommendations for
  policy and practice
• Identification of research gaps

4
The Clinical Guide and Community Guide Are
Complementary

5
The Clinical and Community Guides Are
Complementary
Individual level Clinical settings Delivered by
healthcare providers Screening, Counseling,
etc.
Clinical Guide (USPSTF Recommendations)
Group level Health system changes
Insurance/benefits coverage Access
to/provision of services Community,
population-based Informational (Group
Education, Media) Behavioral, Social
Environmental Policy Change
Community Guide (TFCPS Recommendations)
6
Community Guide (CG) Topics
7
Community Guide Outputs

• gt200 systematic review findings completed
• Family of Products
• Book
• First book published in January 2005
• www.thecommunityguide.org, Oxford Press
• Publications
• Web site
• www.thecommunityguide.org

8
Brief History of the Community Guide

• 1986
• First release of Clinical Guide
• Late 1980s Mid 1990s
• Could there be a comparable guide for public
  health?
• 1996
• Department of Health and Human Services
  established the Community Guide
• As a resource for all of HHS, and for public
  health
• Housed at CDC
• Established a Task Force to direct its work

9
Task Force on Community Preventive Services (Task
Force)

• Nonfederal, independent, rotating
• Internationally renowned experts in public health
  research, practice, policy
• Appointed by CDC director
• Roles
• Oversee topic and intervention priority setting
• Participate on individual review teams
• Make recommendations for policy, practice,
  research, especially for the U.S.
• Not a federal advisory committee

10
Community Guide Intended Uses

• To inform decision making around
• Practice
• Policy making
• Research
• Research Funding

11
Community Guides Approach to Developing and
Disseminating Evidence Based Recommendations

• By actively engaging in conducting and
  disseminating the systematic review those who are
  expected to be the users and beneficiaries of the
  research, it is more likely the findings and
  recommendations will be relevant to their needs

12
So Whose Participation Do We Seek in our
Systematic Reviews?

• Who is to be affected by the research results?
  Who are the intended users?
• Health departments
• Practitioners
• Policy makers
• Professional organizations
• Nongovernmental organizations
• Employers
• Representatives of minority or special
  populations
• Researchers
• Research funders

13
Participants in the Community Guide

• Official Liaisons
• gt25 federal agency and organizational
• Provide input into prioritization, reviews,
  recommendations
• Recommend, find participants for review teams
• Participate on review teams
• Participate in dissemination, translation
  especially to their constituents

14
Official Federal Agency Liaisons

• Agency for Healthcare Research and Quality
• National Institutes of Health
• Office of Disease Prevention and Health Promotion
• Department of Veterans Affairs
• Health Resources and Services Administration

• Substance Abuse and Mental Health Services
  Administration Indian Health Service
• United States Navy Medicine
• United States Air Force
• United States Army

15
Official Organizational Liaisons

• American Academy of Family Physicians
• American Academy of Nurse Practitioners
• American Academy of Pediatrics Bright Futures
  Education Center Advisory Committee
• American Academy of Physician Assistants
• Americas Health Insurance Plans

• American Public Health Association
• American College of Preventive Medicine
• Association of Schools of Public Health
• Association for Prevention Teaching and Research
• Institute of Medicine

16
Official Organizational Liaisons (contd)

• Association of State and Territorial Health
  Officials
• National Association of County and City Health
  Officials
• National Association of Local Boards of Health
• Directors of Health Promotion and Education

• Public Health Foundation
• Society for Public Health Education
• Center for the Advancement of Health

17
Participants in the Community Guide

• Stakeholders, partners
• For specific topics, reviews
• Participate on review teams
• Provide input to Task Force on topic
  prioritization, formation of recommendations
• Participate in dissemination, translation

18
Participants in Individual Reviews

• Coordination Team
• Coordinating scientist (Community Guide)
• Fellows, abstractors (Community Guide)
• Subject matter experts
• From CDC, other federal agencies,
  academia, practice, policy settings
• Task Force member(s)
• Liaison(s)
• Consultation Team
• Subject matter experts
• Community Guide Staff

19
Prioritization of Community Guide Work
20
Priority List - 1998

• Task Force Considered a Range of Factors
• Burden
• Preventability
• Related initiatives
• Usefulness of the package

21
Priority List - 1998

• Decision
• Organize the Community Guide by topics rather
  than by individual interventions
• Output systematic reviews on sets of related
  interventions
• Rationale
• Economies of scale from developing extensive
  expertise, partnerships, support within a topic
• Practitioners, policymakers often interested in
  choosing from a menu of effective interventions
  within a topical area, to best meet their context
  and situation
• But, also permitted specific interventions,
  funded reviews
• Proposed, requested by partners
• To meet immediate needs

22
Note on Task Force Deliberations

• Task Force meetings are public attended by
• Official Liaisons
• Community Guide staff
• Other CDC staff
• Other stakeholders, interested parties

23
The Guide Process From Topic to Intervention
Review(s)
Topic is Selected
Task Force
24
The Guide ProcessThe Team
Topic is Selected
Staff Recruits Teams
Task Force
Coordination Team
Consultation Team
25
The Guide Process Team Decisions
Topic is Selected
Staff Recruits Teams
Task Force
Coordination Team
Team Develops Priority List of Interventions
Consultation Team
Team Decides Breadth vs Depth
Team Develops Conceptual Approach
26
The Guide Process Task Force Input
Topic is Selected
Staff Recruits Teams
Team Presents To Task Force
Task Force
Coordination Team
Team Develops Priority List of Interventions
Consultation Team
Team Decides Breadth vs Depth
Team Develops Conceptual Approach
27
Prioritization 2006ff Our Ultimate Goal is a
Work List

• Reflect Task Force priorities for future work
• With input from Liaisons, other partners,
  stakeholders
• Identify issues to work on next
• As staff and resources become available
• As stakeholders express interest in
  participating, supporting the review
• Identify issues that might be developed with
  additional funding or resources

28
We Remain Open to FundedReview Opportunities
Priority Work List
Funded Review
29
We Have to Update Existing Reviews

• Updates Priority List
• VPD
• PA
• Tobacco
• Diabetes
• .

Priority Work List
Funded Review
30
The Task Force Retains a Place for Special
Projects

• Updates Priority List
• VPD
• PA
• Tobacco
• Diabetes
• .

• Specials Priority List
• Cross-cutter
• Big Intervention
• Dissemination
• ..
• ..

Priority Work List
Funded Review
31
We Also Need a Priority List of New and Continued
Public Health Topics

• Updates Priority List
• VPD
• PA
• Tobacco
• Diabetes
• .

• Specials Priority List
• e.g.,
• Cross-cutter
• Big Intervention
• Dissemination
• ..
• ..

• Priority Work List
• e.g.,
• New Topic
• Expanded Topic
• Update VPD
• Funded Review
• Specials Taxes
• .
• .

• Topics Priority List
• New
• Expanded

Funded Review
32
Step 1 Stakeholder Survey

• Request for additional topics, interventions and
  supporting documentation
• Letter sent to 200 stakeholders in May 2006
• Directors, key scientific staff of CDC programs
  (62)
• Official Liaisons
• Present, past Task Force members
• Other partners and stakeholders

33
Responses from Stakeholders

• Varied in breadth, type of topic, intervention
  proposed
• Land use planning
• Disaster response
• Interventions to increase the dietary intake of
  magnesium
• Interventions to improve the health of older
  adults
• E-Health Communications interventions
• Varied in level of detail provided

34
Step 2 Compile List of Candidate Topics

• Combined topics from original prioritization in
  1998 with new recommendations
• Grouped like interventions into topics,
  highlighting recommended interventions
• List of gt100 candidate topics

35
Challenges Moving Forward

• What should be the organizing structure?
• Disease, condition, risk factor?
• Gender, life stage?
• Women, older adults
• Health protection, health systems issues?
• Poverty, health disparities?
• Considerations
• Move to life stage, health protection, social
  issues
• CDC goals, Healthy People 2020 plans, WHO UN
  Millenium Development Goals
• Current stakeholder needs, public health funding

36
Challenges Moving Forward

• Organizing Structure - Solution
• Four key groupings of topics
• Changing risk behaviors
• Reducing specific diseases, injuries, and
  impairments
• Addressing social, environmental, and ecosystem
  challenges
• Life stages, health protection, and population
  health
• Cross-referenced wherever appropriate

37
Challenges Moving Forward

• Broad vs. narrow focus of topics
• Nutrition vs. breastfeeding
• Infectious diseases vs. tuberculosis
• Solution
• Kept overall topics and focused topics separate
• Focused topics co-located with overall topics in
  tables, other documentation produced
• Ability to select overall topics, focused topics,
  or both was retained through all voting,
  qualitative discussion, grouping, and ranking

38
Product of Step 2

• Master topic list gt100

39
Step 3 Compile Information to Enable Ranking of
Topics

• Challenge
• Providing information for comparison across
  diseases, risk factors, systems, life stage
  issues
• Format
• For each topic (overall and focused)
• Compiled key information in consultation with
  subject matter experts internal and external to
  CDC
• Sent questionnaire to Program Directors at CDC on
  topics related to their areas of focus
• For expanded topics, also included
• List of completed intervention reviews
• List of expanded review options identified by the
  Coordinating Team

40
a) Information Compiled for Each Topic

• Burden, prevalence
• Mortality
• Morbidity
• Subpopulations with higher burden
• Costs
• Other measures
• Potential partners for reviews
• Related Healthy People 2010 objectives
• Related reviews by USPSTF
• IOM reports
• Other reports, reviews, recommendations
• Other notes (e.g., potential overlap)

41
b) CDC Program Directors Questionnaire

• For assigned topics, others they wish to add
• Level of interest in this topic by
• CDC
• Public health field
• Level of priority of the topic by
• CDC
• Other federal entities
• How much influence/leverage could CG
  recommendations have on
• CDC?
• Public health field?

42
b) CDC Program Directors Questionnaire

• For assigned, added topics (contd)
• Their opinion re potential to reduce the public
  health burden (preventability) for the topic
• Additional question
• Knowing the Community Guide cant conduct reviews
  on every topic, what are the 3-5 most important
  topics they should address?

43
b) CDC Program Directors Questionnaire

• Response rate
• (100)
• Several cross-cutting programs declined to rate
  due to ongoing prioritization exercises
• Summary of responses
• For each topic
• Number of CDC programs that rated
• Names of CDC programs that rated
• For each question (e.g., level of interest within
  CDC)
• Mean, standard deviation
• For new topics identified by Questionnaire
• Compiled information on burden, partners, etc.

44
Initial Re-Prioritization of Topics Task Force
Meeting, Fall 2006

• Goal for meeting a rank-ordered list of topics
• Ranking completed by Task Force (TF)
• With input from and discussion with all other
  attendees

45
Initial Plan
Initial Vote
Consensus Discussion
High Priority Tier of Topics (n25)
Rank Vote Within High Priority Tier
Total List of Considered Topics (n111)
Medium Priority Tier of Topics (n50)
Final Consensus
Additional Information
Low Priority Tier of Topics (rest)
46
Mid-Course Modifications
Agree on criteria for voting
111 Topics
1 2 3 4 5 . . . . 111
Initial Vote
Preliminary Rank Order for all 111 topics
47
Key Criteria Agreed Upon for Voting

• Overall attributable burden
• Preventability, attributable fraction
• Leverage
• Is evidence-based guidance currently available?
• Avoid duplication
• Ability to cover a reasonable portion of the
  topic
• Overlap, synergy with the Clinical Guide
• NOT
• Funding or fundability

48
Voting Process Resulting Rankings

• Topics displayed on the screen one by one
• Voted using Perception Analyzer keypad
• 1High priority
• 2Medium priority
• 3Low priority
• Screen advanced once all TF members voted
• Discussion during voting only for clarification
• Produced ranking of 111 topics
• By mean of Task Force members votes
• Also provided frequency of high, medium, low
  priority votes

49
Initial Rank Order Led to (Lots and Lots of!!!)
Consensus Discussion
1 2 . . 25 or so
1 2 3 4 5 . . . . 111
Initial Vote
26 ish 27 28 . 50 or so
51 62 . . 111
50
We Alerted Task Force During Consensus Discussion
of

• Situations where broad topics (and all focused
  topics within it) dropped down in the voting
  process because of split votes across the focused
  topics.
• Topics with overlap dropping down due to split
  voting
• E.g., Reducing medication errors is important to
  consider for a review
• Task Force member A votes for Topic 59 Drug and
  Biologics Safety
• Task Force member B votes for Topic 80 Medical
  Care Safety

51
Consensus Discussion
1 2 . . 25 or so
Move higher ranked Topics Down
Consider if wish to group focused topics, exclude
too broad topics
26 ish 27 28 . 50 or so
Move lower ranked Topics Up
51 62 . . 111
Consider to exclude from further discussion?
52
End of Task Force Meeting
High Priority Tier of Topics 25 or so Loosely
ranked
Medium Priority Tier of Topics
Low Priority Tier of Topics
53
Subsequent Steps

• Formation of Task Force Subcommittee on
  Prioritization to
• Continue reviewing prioritization issues and
  processes
• Prioritize and assess processes for updating
  existing reviews

54
Prioritization of Interventions for Systematic
Review within Topics
Topic is Selected
Staff Recruits Teams
Team Presents To Task Force
Task Force
Coordination Team
Team Develops Priority List of Interventions
Consultation Team
Team Decides Breadth vs Depth
Team Develops Conceptual Approach
55
Prioritization of Interventions for Systematic
Review within a Topic

• Coordination Team subject matter expertise
• They consider
• Degree of interest in the interventions by the
  field
• Interventions where effectiveness entirely
  uncertain
• Potentially effective interventions that are
  underutilized
• Potentially ineffective interventions that are
  overutilized
• Team ranks interventions using a formal Delphi
  process
• All team members receive equal vote

56
For More Information

• Shawna L. Mercer, MSc, PhD
• Community Guide Director
• SMercer_at_cdc.gov

The findings and conclusions in this presentation
are those of the presenter and do not necessarily
represent the views of CDC.