CTSA Evaluation Steering Committee An Overview

1
CTSA Evaluation Steering Committee An Overview
Harold Pincus, Columbia Chair, CTSA Evaluation
Steering Committee Lori Mulligan, NIH
Coordinator CTSA Evaluation Steering Committee
2
Purpose

• Coordinates institutional evaluation efforts with
  the national evaluation of the CTSA program.
• Provides a forum to exchange information about
  institutional evaluation approaches, challenges,
  and progress.

3
Evaluation Steering Committee Composition
CTSA Representatives PI Liaisons
National CTSA Evaluation Steering Committee
NIH CTSA Evaluation Subcommittee
Independent Evaluators
4
Evaluation Steering Committee Workgroups
5
Institutional Evaluation Workgroups

• Definitions and Measurement (Chair B. Trochim,
  Cornell)
• Identify, prioritize, analyze potential
  definitional constructs (i.e., translational,
  disciplinarity, collaboration, etc)
• Synthesize definitions and measurement approaches
• Prepare evaluation briefs/whitepapers that
  summarize sound measurement approaches, tools,
  and methods
• Work with other workgroups, committees as
  appropriate

6
Institutional Evaluation Workgroup

• Social Network Analysis (Chair J. Rainwater, UC
  Davis)
• Explores SNA as a tool to assess communication
  and collaboration
• Visually depict perceived strength of
  relationships among individual groups,
  identifying strong and weak connections among
  them
• Show changes over time in relationships, which in
  turn can demonstrate how the CTSA program is
  affecting the way individuals or groups are
  working with each other.
• Two SNA Examples
• Relationships among key functions - S. Lurie
  (Rochester)
• Co-authorship /betweenness among obesity
  investigators - S. Johnson (Columbia)

Administrative Supplement Using Social Network
Analysis to Evaluate Translational Collaboration
to synergize with Communications SC proposal
Exploring Web-Based Methods to Maximize Sharing
in the CTSA Consortium 
7
Institutional Evaluation Workgroups

• Shared Resources (Chair J. Segal, Johns
  Hopkins)
• Identify and share evaluation resources (i.e.
  surveys, needs assessments, user satisfaction,
  culture change)
• Create a community of committed individuals
  willing to share instruments as well as processes
• Examples https//www.ctsawiki.org/wiki/display/Ev
  al/SharedResourcesWorkgroup
• IRB Issues and Evaluation (Chair P. Moberg, U
  of Wisconsin)
• Workgroup formed as an outcome from Feb 8
  face-to-face meeting (see next slide)

8
Highlights from Feb 8 Face-to-Face MtgRound
Robin Sessions

• IRB Issues and Evaluation Workgroup
• Chair P. Moberg, U Wisconsin
• Consider possibilities to harmonize across
  institutions
• Explore discussions with other CTSA Steering
  Committees (e.g., Alternate IRB models,
  Regulatory, Clinical Research Ethics)
• Identify ways to achieve greater transparency
  through
• Set of tools to educate IRBs about evaluation
• Example of Key Features that an IRB would review
  for exemption
• Convene a subset of CTSA evaluators to engage
  with American Evaluation Association (Scheduled
  for November 7 at AEA Conference)

9
Highlights from Feb 8 Face to Face MtgRound
Robin Sessions

• The Role of the Evaluator (Facilitator D. Rubio,
  U Pittsburgh)
• Discussed different evaluation approaches
  underway core performance v. program
  evaluation external v. internal consultants
• Identified challenges how best to capture real
  change/transformation how do you evaluate
  leadership and to whom and how are results
  reported
• Consider what role the evaluator has or should
  have with external advisory board

10
Highlights from Feb 8 Face to Face MtgRound
Robin Sessions

• Evaluating Indirect Benefits of the CTSA
  (Facilitator A. Dozier, Rochester)
• Consider economic impact
• Ability to draw different skill sets into C T
  research
• Assess nature of new partnerships and ability to
  attract unexpected collaborators
• New ways to engage communities
• Assess cultural changes
• Generate examples of indirect effects and post on
  Wiki (i.e., Rochester economic impact)

11
Schedule of Upcoming Meetings

• Evaluation Steering Committee
• August 20, 2008, 230 400 ET
• October 15, 2008, 230 400 ET
• November 8 9, 2008, Denver, Face-to-Face
  meeting
• Held in conjunction with the American Evaluation
  Association Annual Conference that will feature
  18 CTSA presentations, 22 CTSA evaluators, and 10
  CTSA institutions
• Standardizing and Operationalizing Definitions
• September 9, 2008, 430 530 ET
• October 7, 2008, 430 530 ET
• October 21, 2008, 430 530 ET
• Shared Resources
• September 11, 2008, 300 400 ET
• October 9, 2008, 300 400 ET
• November 13, 2008, 300 400 ET
• Social Network Analysis
• September 11, 2008, 300 400 ET
• November 20, 2008, 300 400 ET

12
November 8-9, 2008 Face-to-Face Meeting

• Note Please submit agenda topics and suggestions
  for breakout sessions on the Wiki
• https//www.ctsawiki.org/wiki/display/Eval/AddAge
  ndaTopics-Face-to-FaceMeetinginDenver-Novem
  ber8-92C2008
• Meeting Schedule (All times are local Denver
  time, i.e. MT)
• Date Time
• Saturday, Nov 8 630pm 830pm
• Sunday, Nov 9 800am 300pm
• Registration
• Registration will begin at 545pm on Saturday,
  November 8, with light refreshments beginning at
  600pm. Registration will continue into Sunday,
  November 9 and will begin at 715am.
• Meeting Location
• Hilton Garden Inn Denver Downtown
• 1400 Welton Street
• Denver, CO 80202
• Phone [phone]
• Lodging (0.2 miles from the meeting location)

13
Next Steps for New Members

• Join workgroups, recommend forming new ones,
  propose topics for webconferences and meetings
• Participate in August 20 Full Steering Committee
  Webconference
• Attend and help shape agenda for the November 8 -
  9 Evaluation Steering Committee meeting in
  Denver, CO
• Held in conjunction with the American Evaluation
  Association Annual Conference that will feature
  18 CTSA presentations, 22 CTSA evaluators, and 10
  CTSA institutions

14
Evaluation SC Points of Contact

• Steering Committee Chair Coordinators
• Dr. Harold Pincus, CTSA Chair
• E-mail pincush_at_pi.cpmc.columbia.edu
• Ms. Lori Mulligan, NIH Coordinator
• E-mail mulligal_at_mail.nih.gov
• Ms. Huda Aden, Project Manager
• E-mail aden_huda_at_bah.com
• PI Liaisons to Evaluation SC
• Dr. Garret FitzGerald, University of Pennsylvania
• E-mail garret_at_spirit.gcrc.upenn.edu
• Dr. Robert Rizza, Mayo Clinic College of
  Medicine, PI Liaison
• E-mail rizza.robert_at_mayo.edu

15
Evaluation SC Points of Contact

• Workgroup Chairs
• Dr. William Trochim, Chair, Definitions and
  Measurement Workgroup
• E-mail wmt1_at_cornell.edu
• Dr. Julie Rainwater, Chair, Social Network
  Analysis Workgroup
• E-mail jarainwater_at_ucdavis.edu
• Dr. Jodi Segal, Chair Shared Resources Workgroup
• E-mail jsegal_at_jhmi.edu
• Dr. Paul Moberg, Chair IRB Workgroup
• E-mail btai_at_nida.nih.gov

16
Evaluation SC Resources

• CTSAweb.org
• http//www.ctsaweb.org/index.cfm?fuseactioncommit
  tee.viewCommitteecom_ID19
• CTSA Wiki
• https//www.ctsawiki.org/wiki/display/Eval/Evaluat
  ion-Home