British Association

1
Developing evaluation approaches for science in
society initiatives

• Joe Cullen
• Fay Sullivan
• Tavistock Institute

2
Project overview

• Commissioned by OSI in association with ESRC
• To develop an approach/ framework for evaluating
  science in society initiatives
• 2 phases
• Draft evaluation framework
• Pilot framework in real-world environment
  (SETNET)

3
In this presentation we want to

• Focus on learning from phase 1 (Joe)
• the nature of science in society initiatives
• and the broad contexts in which they are situated
• Introduce the framework and its key components
  (Fay)
• Discuss conference participants experiences and
  expectations of evaluation

4
(No Transcript)
5
Personnel in RD
6
(No Transcript)
7
Why the interest?

• Science increasingly important policy driver
• Lisbon goals most competitive global economy by
  2010
• UK policy Science and Innovation Investment
  Framework
• Lagging behind in global RTD investment
• Skills base improving but variable
• Under-representation of women BEMGs

8
The Big Issues

• Public concerns with science shift from love
  affair (Victorian) to current fear and loathing
• Science not cool decline in science subjects
  and science careers
• Not intelligible and not understood lexicon of
  mystery
• No real business or cost effectiveness model
• No science entrepreneurs (c.f. SETNET
  Ambassadors)

9
The social construction of science

• No single universal scientific truth
• Shaped by prevailing synchronic and diachronic
  processes
• Dystopian view (Habermass Heidegger)
• Dialogic reflexivity (Giddens)
• Civilisational choices (Feenberg)
• Folk devils and moral panics (Young) GM BSE

10
Civilising Choices
18th C Spanish Flintlock Blunderbuss Pistol
11
Conquistadores

• 1532 Pizarro destroys 80,000 army of Atahuallpa
  with 168 Spanish soldiers
• Led to economic, social and cultural decimation
  of most powerful New World civilisation
• 21st century legacy
• Peru GDP per capita 2,300
• Spain GDP per capita 20,150
• Contribution to globalisation problems

12
Science in Society models

• Deficit model and construction of the scientific
  citizen (Irwin)
• OST/Wellcome research - six clusters, linked to
  personality types
• Contest of meanings between academic,
  industrial, bureaucratic, and civic groups
  (Elzinga and Jameson)
• Societal learning public demanding more say
  in construction of knowledge

13
How science is constructed

• Social structures economy and culture
• History (e.g. Great Wars)
• Individual personality
• Power relations
• Communities of practice and constituencies of
  knowledge
• Context tacit and explicit knowledge
• New forms of dialogue and process Knowledge
  Society

14
Scientific Knowledge Creation
Experts Evidence-based explicit knowledge
Communities of Practice Practices
Life-Worlds Ethnographies
15
How Knowledge really works
Communities of practice
Expert constituencies
Life worlds
16
Collaborative knowledge

• IKEA fiffiga folket scheme (ingenious
  people) to develop creative ideas from ordinary
  people
• OhMyNews - South Korea - 38,000 citizen
  reporters distributed knowledge gathering
  resource
• Adbusters - citizen capitalism- ethical
  trainer
• MySpace
• Wikipedia
• De.Licio.us
• Flickr

17
(No Transcript)
18
Why is understanding how science is constructed
important?

• the object of the evaluation (what is the unit
  of analysis to be evaluated)
• the purposes and scope of the evaluation (e.g.
  awareness education skills development
  careers)
• the range of stakeholders that need to be
  involved (and power dynamics)
• the type of evidence that will be accepted
• the questions that need to be asked and the
  criteria that need to be used to draw conclusions

19
Types of SiS initiative

• Large scale awareness-raising campaigns
• Public participation
• Interactive events (outreach theatre
  demonstrations)
• Education and training
• Ongoing profile-raising
• Targeted access and inclusion actions
• Policy actions
• Horizontal and supporting actions
  capacity-building
• Operational Reviews

20
Evaluation problems

• Scientific knowledge evolving contested
• Different constructions
• Different purposes - awareness subject choices
  career transitions investment
• Different power structures and stakeholder groups
• Change-focused and prospective
• Knowledge society new social forms new forms of
  knowledge
• Deficit model inadequate societal learning
  more participatory more collaborative and
  developmental
• Lack of published knowledge base on
  domain-specific evaluation methods

21
Evaluation solutions

• Framework captures complexity and makes it
  manageable
• To decide how to evaluate, need to answer what
  are you evaluating and why?
• Dont think about methods until youve answered
  these questions
• Scoping/ preparation work is key

22
What are you evaluating?

• Tools to map initiative characteristics and
  contexts
• Distal dimensions e.g.
• Broad purpose of initiative
• Cultural logic
• Policy agenda and focus
• Proximal dimensions e.g.
• Level of analysis
• Scale
• timeframe

23
What are you evaluating? mapping tools

• Excerpt from distal dimension mapping tool
• Excerpt from proximal dimension mapping tool

24
Why are you evaluating?

• Different stakeholders have different priorities
• Common evaluation purposes e.g. to
• Generate knowledge
• Develop the initiative
• Improve implementation

25
How should you evaluate?

• Answering what and why questions create the
  clarity and conditions to decide on evaluation
  questions and criteria
• Then you can think about (multiple) methods
• An (oversimplified) example
• purpose implementation
• evaluation question is initiative being
  delivered effectively?
• approach formative
• method semi-structured interviews with deliverers

26
Discussion

• Some of our questions
• Does what weve said resonate with you?
• What is your experience and expectation of
  evaluation?
• Is an evaluation framework helpful? If yes, what
  would you want it to look like?
• Some of your questions