The Politics of Policy Appraisal and Evaluation

1
The Politics of Policy Appraisal (and Evaluation)

• Andrew Jordan, Claudio Radaelli and John Turnpenny

2
Why these topics now?

• Changes in the policy world
• Decentralised governance (indicators, charters)
• Post ideological (political?) policy making
• Evidence (or science based) based policy making
  (what matters is what works)
• Better/meta regulation
• Manifestations (institutionalisations) of these -
  RIA, IA also NICE, agencies etc.
• Evolving academic foci
• Evaluation - a mid-life crisis? Appraisal?
• Evidence (knowledge) use when, what, why, how?
• Knowledge based theorising (Heclo, Haas,
  Sabatier, Hall, Radaelli et al.)

3
Overview

• Defining appraisal and evaluation (research)
• What are the politics of appraisal/evaluation?
• Mapping out the field(s)
• Structure of the rest of today/Tuesday (John)

4
Defining appraisal and evaluation
5
Appraisal and evaluation

• Policy appraisal is
• that family of ex ante techniques and
  procedures. that seek to inform decision makers
  by predicting and evaluating the consequences of
  various activities according to certain
  conventions (Owens et al. 2004 1944).
• Policy evaluation is
• Concerned with undertaking ex post assessments of
  the strengths and weaknesses of public programs
  and projects
• It is about measuring and assessing the value or
  social worth of these things.

6
Defining the politics of.
7
Defining the politics of.

• if there are concepts that are more difficult to
  define than evaluation and politics, I have
  not seen them.
• (Dennis Palumbo, 1987 15)

8
The core meaning of politics

• Politics is what governments do (polis Greek
  for city-state) Eastons authoritative
  allocation of values
• Politics is the broader social process of
  decision making Laswells Who Gets What,
  When and How?
• All events and processes and practices which
  occur in the social sphere have the potential to
  be political and, hence, should be amenable to
  political analysis (Hay 2003 3).

9
Different purposes of policy analysis

• A process of multi-disciplinary inquiry designed
  to create, critically assess, and communicate
  information that is useful in understanding and
  improving policies (Dunn 2004 2).
• Improving - analysis for policy - speaking the
  truth to power, evidence use, EBPM etc.
• Understanding - analysis of policy policy
  analysis as an academic discipline (e.g. the
  stages model)

10
Evaluation and the policy cycle
In Palumbo, D. (1987) Politics and Evaluation.
In D. Palumbo, (ed.) The Politics of Program
Evaluation. Sage, London.
11
Policy evaluation (research)

• Rational roots evaluators as technicians to
  the powers that be (Weiss)
• Introduction of business/management approaches
  the audit state
• The postmodern assault the evaluator as
  orchestrator
• The realist backlash realistic evaluation.
  now what?

12
Policy appraisal (research)

• Technical-rational roots
• Initial development in the USA (EIA and RIA)
• Increasing focus on the (mis)use, effectiveness
  and/or diffusion of appraisal procedures//tools
• Emerging interest in appraisal as part of the
  policy/political process
• But the technical-rational roots remain strong

13
Mapping out the fields.
14
Ex ante assessment
15
Ex ante assessment
ENV SD EIA, SIA, SEA Cashmore, Owens, Nilsson,
Russel, Sheate, Thissen, Jacob, Jordan,
Turnpenny, von Raggamby
BR RIA Regulatory law Radaelli,
Meuwese Scharrenborg, Ogus
16
Ex ante assessment
Evaluation (Principally health, education,
criminal justice, social care) Palumbo, Weiss,
Patton, Stame, Sanderson, John, Pawson,
Balloch, Stern, Hansen
ENV SD EIA, SIA, SEA Cashmore, Owens, Nilsson,
Russel, Sheate, Thissen, Jacob, Jordan,
Turnpenny, von Raggamby
BR RIA Regulatory law Radaelli,
Meuwese Scharrenborg, Ogus
17
EBP Nutley, Parsons, Sanderson Boswell
Ex ante assessment
Appraisal policy Nilsson, Owens, In t Veld,
Sheate, Russel, Martinuzzi, Thissen Vasilescu
Evaluation (Principally health, education,
criminal justice, social care) Palumbo, Weiss,
Patton, Stame, Sanderson, John, Pawson,
Balloch, Stern, Hansen
ENV SD EIA, SIA, SEA Cashmore, Owens, Nilsson,
Russel, Sheate, Thissen, Jacob, Jordan,
Turnpenny, von Raggamby
BR RIA Regulatory law Radaelli,
Meuwese Scharrenborg, Ogus
18
STS Sarewitz,Ravetz
Integrated Assessment
EBP Nutley, Parsons, Sanderson Boswell
Ex ante assessment
Appraisal policy Nilsson, Owens, In t Veld,
Sheate, Russel, Martinuzzi, Thissen Vasilescu
Evaluation (Principally health, education,
criminal justice, social care) Palumbo, Weiss,
Patton, Stame, Sanderson, John, Pawson,
Balloch, Stern, Hansen
ENV SD EIA, SIA, SEA Cashmore, Owens, Nilsson,
Russel, Sheate, Thissen, Jacob, Jordan,
Turnpenny, von Raggamby
BR RIA Regulatory law Radaelli,
Meuwese Scharrenborg, Ogus
19
Key similarities

• They are both
• Highly policy focused informing governmental
  policy makers and practitioners
• Seeking to study quite specific (but linked)
  stages of the policy process (Radaelli, 1995
  163)
• Descended from rational-technocratic approaches
• Underpinned by positivistic assumptions about
  knowledge and methods i.e. we can evaluate and
  appraise in an objective fashion.
• Surprised that evidence is (often) not used as
  expected
• Involve analysis for and of the policy process
• Framed by/interested in changing patterns of
  governance (the evidential turn) (Taylor 2005
  601).

20
Common concerns evidence use

• Instrumental and conceptual uses, eventually
  challenging policy paradigms (e.g.
  enlightenment)
• Tactical and strategic (mis)uses
• Process and output uses
• Planned and unplanned uses
• Formal and informal uses.
• From Nutley et al (2007) Patton (1987 111).

21
Some important differences

• Appraisal is more about recommendation and
  prescription, whereas evaluation is more about
  assessing the value or worth of outcomes
• They serve different (policy) masters
• Evaluation has done more to investigate the (non)
  utilisation of knowledge
• In evaluation research, there are top down and
  bottom up approaches what about policy
  appraisal?
• The evaluation community has accepted that
  evaluation is politicised the issue is what to
  do about it?

22
If they are politicised, then what?

• Analysis of - policy evaluation is nothing but
  the continuation of politics by other means
  (Bovens et al. 2006 321).
• Analysis for - Knowing that political
  constraints and resistance exist is not a reason
  for abandoning evaluation research rather it is
  a precondition for usable evaluation research
  (Weiss, 1987 48).

23
Examples of the politics of

• Politics is what governments do
• Much appraisal/evaluation occurs within
  government it creates a statement
• Politics frames both it shapes and triggers
  them (failure?)
• Some are expressly commissioned to support a
  particular policy (bureaucratic politics!)
• Their findings are fed in and hence become part
  of the policy process
• They have sponsors and supporters they are
  produced by actors with opposing imperatives
  (Palumbo 1987 19)
• Politics as the broader social process of
  decision making
• They involve value judgements which are
  essentially political positions on an issue
• Non-states actors are often involved (brought
  in?)
• Knowledge is power - evaluation has
  accountability and lesson drawing implications
  that highlight winners and losers
• They operate within broader policy frames and
  paradigms which indicate what is important and
  what needs doing

24
The Politics of Policy Appraisal (and Evaluation)

• Andrew Jordan, Claudio Radaelli and John Turnpenny

25
Aims of the workshop

• Consolidate the state of the art
• Novel connections
• Identify new and promising research needs
• Bottom-up process
• Set the ground

26
The structure of to-day

• Themes
• Speakers and discussion

27
Theme 1The aims and objectives of appraisal

• Why does society appraise/evaluate (the
  evidential turn) and how?
• What frames/shapes these activities?
• What problems do they seek to address
  (problem-definition analysis)?
• The different contexts of appraisal/ evaluation -
  top down and bottom up approaches

28
Theme 2 Underlying Theories
29
Theme 2 Underlying Theories

• What are the embedded theories of the policy
  process?
• What is the dominant notion of public
  administration (NPM, learning etc.)?
• What are the underlying ontologies and
  epistemologies?
• What are the theories of collecting, weighing and
  using evidence?
• What are the models of politics (cabinet
  decision-making, relationship between bureaucrats
  and politicians, etc) that inform designs?
• Links into the broader debate about governance

30
Different models of use

• knowledge driven model
• problem solving model
• interactive model
• political model
• tactical model
• enlightenment model
• research as an intellectual enterprise
• From Carol Weiss (1975 1979 etc.)

31
A typology rather than a list?

From Nutley, Walter and Davies (eds) (2007)
Using Evidence. Polity Press, Bristol.
32
Theme 3 Appraisal/evaluation in practice

• Looking at outcomes do they produce better
  regulation or SD?
• Does it inform policy making, in the short (often
  not) and longer term (possibly?)?
• Does one draw on the other (longer term
  learning?)
• Winners and losers who gets what.?
• The use of knowledge in politics symbols,
  emotions and constructions of findings in
  discourse

33
Theme 4 Future directions for research and
practice

• What sorts of processes and contexts promote
  usable knowledge?
• What works - the barriers and enablers
  literature?
• Developing new conceptual definitions and
  categorisations
• Levels of analysis tools? procedures?
  institutions? knowledge?
• New politically attuned methods of evaluation and
  appraisal
• Temporal analyses (what time frame?)

34
Practicalities

• Lunch and dinner
• Travel expenses claims Please return with
  receipts to
• Dawn Turnbull, CSERGE, University of East Anglia,
  Norwich, NR4 7TJ

35
Tuesday

• Process
• Outcomes journal/book/seminars?