Assessing the impact of research and KTE activities

1
Assessing the impact of research and KTE
activities

• Sandra Nutley

2
Agenda for this session

1. Key approaches to assessing the use and impact of
   research and knowledge mobilisation activities
2. Common challenges and methodological issues
3. Summary of findings from existing assessment
   activities

3
Why assess research impact?

• Addressing accountability
• Assuring value for money
• Setting priorities
• Assisting learning
• Improving outcomes
• Summative or formative purposes?

4
1. Key approaches to assessing research impact
Impact of research
Single studies
Research programmes
Systematic reviews
Policymakers
Target KTE interventions
Practitioners
Media
Centres promoting research use
Organisational research use
Research use in user communities
Impact of initiatives to increase research use
5
Forward tracking from research projects and
programmes to use and impact

• Two common methods
• Describe and quantify impacts of research - E.g.
  Payback model 5 categories of possible impact
• Knowledge production
• Research capacity building
• Policy or product development
• Sector benefits
• Wider societal benefits
• Describe and map networks and flows of knowledge
  and the effects of any interactions

6
Tracking back from decisions or behaviours to
research influences on these

• Bibliometric studies of policy documents and
  practice guidelines
• User panels surveys and workshops
• Ethnographic studies of communities of practice
• Economic impact evaluations estimate economic
  impact of a policy and estimate extent of
  research influence on that policy

7
What methods are used in practice? Systematic
review by Boaz et al 2008
8
Evaluations of initiatives to increase research
impact

• Standard programme evaluation strategies
• Qualitative investigation of processes and
  perceived effects
• Action research
• Experimental and quasi-experimental methods
• Importance of strong theories of change models
  of KTE to design and test interventions
• Need shared taxonomy of KTE strategies and
  interventions

9
KTE models

• Ward et al 2009
• 63 different theories or models of KTE
• Distilled down to 5 components of the KTE process
• Connection between them is seen as interactive
  and multi-directional
• Aim of framework is to provide a foundation for
  gathering evidence about knowledge into action
  processes

10
Context Barriers/Supports
11
Taxonomies of KTE strategies

• E.g. Mitton et al 2007 8 key strategies
• Face-to face exchange between researchers and
  users
• Education sessions for users
• Networks and communities of practice
• Facilitated meetings between researchers and
  users
• Interactive, multidisciplinary workshops
• Capacity building in user organisations
• Web-based information, electronic communications
• Steering committees for research
  projects/programmes

12
RURU taxonomy

• Intervention types and underlying mechanisms,
  with focus on the latter
• Five key mechanisms
• Dissemination
• Interaction
• Social influence
• Facilitation
• Incentives and reinforcement

13
2. Common challenges and methodological issues

• What types of use/impact are of interest?
• When to assess impact?
• Importance of context assessing actual or
  potential impacts?
• Dealing with attribution and additionality
  constructing a convincing impact narrative
• Getting away from linear models research
  use/impact

14
What uses/impacts are of interest?

• Instrumental, conceptual or symbolic research
  use?
• Reach/awareness, impact on behaviour service
  delivery, outcomes for service users?
• Positive and dysfunctional consequences?

15
When to assess impact?

• Impact often occurs far down the line, but impact
  trails grow fainter over time

16
Tools for analysing context

• Accounting for the difference between actual and
  potential impacts
• E.g. Conducive policy environments - research
  more likely to be used when information, ideology
  and interests coincide and institutions can
  support developments (Weiss)

17
Constructing a convincing impact narrative

• Dealing with attribution the potential of
  contribution analysis

18
Slide source Wimbush 2010
19
Slide source Wimbush 2010
20
Slide source Wimbush 2010
21
Conclusions on methods

• Case study methods to take account of differing
  types of research and contexts for impact
• Combination of quantitative and qualitative
  methods and indicators within case study approach
• Need for research impact theory/model to guide
  data collection and analysis
• Limited benefits from studying the impacts of
  individual research projects
• Dangers of generalising from case sampling
  because distribution of research impact is likely
  to be uneven

22
3. Summary of findings from the UK ESRCs impact
assessment activities

• The most important drivers of impact are
• Established relationships and networks with user
  communities
• Involving users at all stages with research
• Well-planned user-engagement and KTE strategies
• Portfolios of research activity that build
  reputations with research users
• Good infrastructure and management support
• The involvement of intermediaries and knowledge
  brokers as translators, amplifiers, network
  providers

23
RURUs conclusions on generic features of
effective KTE practices

• Research must be translated - adaptation of
  findings to specific policy and practice contexts
• Enthusiasm- of key individuals - personal contact
  is most effective
• Contextual analysis - understanding and targeting
  specific barriers to, and enablers of, change
• Credibility - strong evidence from trusted
  source, inc. endorsement from opinion leaders
• Leadership - within research impact settings
• Support - ongoing financial, technical
  emotional support
• Integration - of new activities with existing
  systems and activities

24
Implications for future research on research use
and impact

• Much that we still need to know
• We need to move away from
• Poor documentation and under-evaluated KTE
  activities
• Studies that focus only on the instrumental use
  of research
• An assumption that research is used and applied
  mainly by individual practitioners
• Studies that result only in a now familiar
  listing of barriers and enablers, especially
  where these are the barriers/enablers experienced
  by individual practitioners

25
Key messages

• Need to understand more about how research-based
  knowledge enters and flows through diverse
  policy, organisational and practice settings
• Need realistic assumptions about the nature and
  processes of research use and impact these are
  many and complex
• No single model of research use is likely to be
  sufficient for all situations when impact is to
  be assessed
• Need to make choices about where and how to look
  for use and impact based on the purpose of the
  assessment

26
References

• Main reference
• Davies HTO and Nutley SM (2008) Learning More
  about How Research-Based Knowledge Gets Used
  Guidance in the Development of New Empirical
  Research, Working Paper for the WT Grant
  Foundation, New York
• Other useful references
• Boaz et al (2008) Assessing the impact of
  research on policy A review of the literature,
  Kings College London/ PSI
• ESRC (2009) Taking Stock A summary of ESRCs
  work to evaluate the impact of research on policy
  and practice, http//www.esrc.ac.uk/_images/Taking
  20Stock_tcm8-4545.pdf
• ESRC (2010) Branching out New direction in
  impact evaluation from the ESRCs Evaluation
  Committee http//www.esrc.ac.uk/_images/Branching
  20Out_tcm8-14881.pdf
• Meagher L, Lyall C and Nutley S (2008) Flows of
  knowledge, expertise and influence a method for
  assessing policy and practice impacts from social
  science research Research Evaluation 17(3)
  163-173
• Ward et al (2009) Developing a framework for
  transferring knowledge into action a thematic
  analysis of the literature Journal of Health
  Services Research and Policy 14(3) 156-164

27

• For further information
• The following three slides summarise the main
  research questions identified by Davies and
  Nutley (2008) when they were asked to define the
  emerging research agenda on research use and
  impact for the WT Grant Foundation. These
  questions are categorised under the three
  headings of
• Knowledge source, presentation and integration
• Context and connections
• Strategies and process

28
Knowledge source, presentation and integration

• What models of research supply and synthesis
  might better support knowledge integration by
  potential users?
• How do different kinds of messaging and
  messengers affect the use of research knowledge?
• What is the role of the web in providing access
  to existing research?
• To what extent are policymakers and service
  managers conducting their own in-house research
  using administrative and local data?
• How is new knowledge integrated into current ways
  of thinking and models of practice?

29
Context and connections

• What communities are involved in the field of
  interest? How are these connected?
• Are policy and practice communities networked and
  does research-based knowledge flow across these
  networks?
• How, where and under what circumstances do
  practitioners source new knowledge?
• What models of research brokerage or intermediary
  activities have the best potential for fostering
  research use?
• How can lay people, service users, and others
  contribute more fully to evidence-informed
  discussions?
• How are education and CPD connected to and
  supportive of knowledge accumulation and
  integration?

30
Strategies and processes

• Do different models of research use co-exist in
  different practice settings?
• What knowledge management strategies are used
  across the sector?
• What models of push, pull and linkage-exchange
  are in place? How are these evolving and
  connecting?
• What strategies for increasing research use and
  impact have the best evidence in support of them,
  and how might they be tested to consolidate our
  knowledge base in this area?
• How can we get new knowledge on the
  research-action process to influence the future
  actions of researchers, funders, intermediaries,
  policy/decision-makers, and practitioners/end
  users?