Rani Mahey Project Officer

1
Welcome

• Rani Mahey Project Officer
• Tel [phone]
• E-mail rani.mahey_at_regenwm.org

2
House Keeping
3
Why this seminar ?

• Egan Review
• Multi-disciplinary knowledge
• Good Practice
• Generic Skills
• We wont make you an expert but..
• Understanding of the key concepts, priorities etc
• Refreshing knowledge
• Demystifying
• Encourage you to learn more.

4
Learning goals

• Developments in evidence-based project
  development
• Understanding appraisal processes from the
  funders perspective.
• Risk and reactions to it - in regeneration.
  Optimism bias, incentive skew and so on.
• Application of development tools (for instance,
  the Logical Framework Approach)

5
What is your role?

• Everybody here is from a different background and
  perspective. Sharing this will add to the
  learning experience for all.
• Everybody has different levels of understanding.
  If you know contribute. If you dont know
  ask!!!!!

6
Speaker introduction

• John Batchelor
• Ian Waddell
• Williamson Joyce Research  

7
Seminar
series

• Designing better projects

8
Welcome. This seminar is about

• Appraisal
• Evidence based project development
• Risk and reactions to it
• Monitoring and evaluation

9
Who are we?

• John Batchelor,
• Ian Waddell
• Williamson Joyce Research
• john_at_wjresearch.com 079 41401818

10
And you are.
11
This session is about

• The process of appraising projects (usually those
  seeking grant money)
• What appraisers say are the most common mistakes
  in applications
• Tips on how you can avoid these mistakes

12
What is project appraisal?

• Process for assessing the merits of a (financial)
  intervention market failure
• Method for prioritising bids in competitive
  funding streams
• Based on organisational and governmental
  standards (the Treasury Green Book)

13
Who does appraisal?

• Civil servants, QUANGO staff (e.g. English
  Heritage), independent appraisers
• Most applications are checked at administrative
  levels first
• Despite standards, it is not an exact science,
  and
• despite appearances, appraisers are
• human

14
Who else should do appraisal?

• People with responsibilities for designing
  projects and making applications

15
Why bother?

• Understanding the process will help you design
  better projects
• Appraisal is a key part of the project lifecycle
  good appraisal will help with later evaluation

16
Project lifecycle
17
The bidding process not perfect

• Most bidding processes are competitive and many
  complain of beauty contests
• Application forms are not always clear and some
  have poor guidance
• Deadlines are often very tight as initiatives
  come and go
• Aim of process is to be fair to all, but feedback
  is sometimes hard to get

18
What appraisers assess

• Extent to which a project is realistic based on
  sound assumptions and deliverable
• Whether it provides additionality or is a
  deadweight project
• Whether the activities are worth the money
  compared to other options (value) often on a
  unit cost basis
• Whether the project meets the aims objectives
  of the fund

19
The appraisal process - problems reported at
application stage

• Applications delivered after the deadline
• Applications in the wrong format
• Details to be supplied later
• Speculative bids that ignore eligibility

20
Why appraisers reject projects our survey said
Part 1

• Cut and paste contradictions
• Meaningless annexes
• Basic mathematical errors
• Gobbledygook and misused buzz words
• Woolly and immeasurable outputs and outcomes
• Unrealistic (irrelevant or unachievable) targets
• Over optimistic timetabling (milestones)

21
Why appraisers reject projects our survey said
part 2

• Insufficient research on client group
• No/weak delivery options
• No/weak risk analysis or contingency
• No/weak succession plan
• Unclear monitoring arrangements

22
How to avoid basic pitfalls

• Unless you have friends in high places, play by
  the rules
• Make sure you understand the funds eligibility
  and application format. If in doubt, ask.
• Set a pre-deadline deadline
• Get a receipt on submission

23
Cut and paste

• Difficult to avoid if many people are writing or
  you legitimately want to use other projects
  ideas
• BUT
• Dont show the joins aim for a logical narrative
• Have someone read it for sense (someone not
  directly involved)
• Proof for maths errors and consistency
• Dont include annexes unless they support the
  case, and reference them

24
Outputs and outcomes
25
Outputs and outcomes

• Outputs Are the services, activities and products
  that organisations deliver
• Outcomes Are all the changes, benefits, learning
  or other effects that happen as a result of what
  is delivered
• In setting targets, use SMART they should be
• Specific
• Measurable
• Achievable
• Relevant
• Time-bound

26
Monitoring

• Appraisers expect monitoring to reflect the
  resources available, but
• Ensure that if you say youll generate 1000 extra
  visits, you say how youll know
• Less tangible outcomes like a better
  understanding of a collection might be measured
  through before and after surveys

27
Capture the project simply without jargon

• Write a short statement about what the project is
  for, and what it will achieve
• Test it on non-experts
• Use it to introduce your application
• Follow this plain, clear, style throughout

28
Short statement - example

• Groovy Gallery has identified through
  research that local primary school children are
  under-represented in its visitor profile. In
  order to address this, we will organise a
  tile-design day for 15 local children in January
  2006, supported by a local artist. The project
  will introduce children to the Gallery and
  support the Arts curriculum. Local schools have
  supported the project. We will provide vouchers
  for the children so that we can identify how many
  subsequently re-visit the Gallery, and monitor
  wider use in 2006 by 5 - 10 year-olds through our
  ticketing system, to judge the effectiveness of
  the project.

29
Do some research

• A basic understanding of the projects
  (potential) client group is vital
• Access to data is now much easier most
  statistics are available on the web
• Use what you collect day-to-day as part of your
  evidence if there are gaps, fill them
• (but do this within the scale of resources bid
  for)

30
Show your working

• In order to arrive at your proposed approach, you
  will have considered a number of options
• Say why you have selected this option, and why
  the others (at least 2) are less good
• Always include a do nothing option, and say
  what the consequences are (this is, after all,
  why you are doing the project in the first place)

31
When and what next

• Making applications, winning grants, and setting
  up projects is a frustratingly slow process
• Allow time from a contract stage to establish the
  project. Employing staff (particularly
  specialists), for instance, takes an age
• Say what will happen after the project is
  completed

32
A chink of light !

• Developing projects is not so hard if you have
  already established good information management
  systems
• Appraisers want to approve good projects it is
  hard for an appraiser to turn down a good project
  because of missing information but they are not
  allowed to guess
• People who understand how to develop projects are
  in short supply, and are much in demand
• Grants do provide the funds for lots of really
  good projects

33
Evidence

• The process of developing a project rationale
  through evidence
• Project development schemas
• Establishing monitoring

34
A bit of history

• 1980s
• From local to central political control
• Public to private sector led
• Social to economic priorities
• From needs led to market led
• total approach to ABIs

35
A bit more history

• 1990s added
• Competitive bidding
• Co-ordination
• Co-operation (partnership)

36
The Third way

• PATs SEU
• Back to needs led
• Decentralisation
• Sustainable regeneration
• Lottery

37
What is a project?
38
Where do projects come from?

• Lobbying/campaigning/rioting
• Policy intervention
• Piloting
• Research
• Copying
• The bath

39
Key project development principle

• Quality must be designed in,
• not inspected out.

40
Project development schemas

• TOWS methodology
• Green book
• Logical Framework Approach

41
TOWS methodology
42
Rationale from Green Book process

• The result if nothing changed, or if there was
  minimal change
• The market situation (e.g. cause of any market
  failure, employment levels)
• Current and projected trends and published
  forecasts (e.g. population, services volume,
  demand, relative prices and costs)
• Potential beneficiaries (and those who may be
  disadvantaged)
• Technological developments and,
• Whether the problem to be addressed changes in
  scope or magnitude over time e.g., effects can
  multiply over generations.

43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
Risk. (Lots of them)

• At risk groups vulnerable people
• Risk assessment employee safety
• Environmental risk floods
• Investment risk credit crunch
• Moral risk disrepute
• Regeneration risks are mainly intervention risks

56
In the news

• House revival scheme 'high risk'
• A 2bn government programme to revive rundown
  housing in central and northern England is
  "radical but high risk", a National Audit Office
  report says.
• The spending watchdog says that demand in the
  "Pathfinder Areas" has risen, but it is not clear
  if that is because of the scheme launched in
  2002.
• BBC 9th November 2007

57
Risk in regeneration

• Risk is inevitable
• Regeneration projects, by definition, are risky
• Risk is a management issue
• The biggest management issue is information
  quality

58
How risky am I?

• 334.64

59
(No Transcript)
60
Frameworks

• Treasury Green Book
• http//greenbook.treasury.gov.uk
• Logical Framework Approach
• www.everychildmatters.gov.uk/resources-and-practic
  e/search/EP00025/

61
Reactions

• Optimism bias
• Incentive skew
• Risk aversion

62
Optimism bias

• demonstrated systematic tendency for appraisers
  to be over-optimistic about key project
  parameters. It must be accounted for explicitly
  in all appraisals, and can arise in relation to
    Capital costs Works duration Operating costs
  and Under delivery of benefits.

63
Incentive skew

• Risks not identified or played down because of
  incentives in grant processes or appraisal

64
Risk aversion

• Avoiding all risk
• in itself, risky?

65
Limiting impact

• Consulting early
• Avoiding irreversible decisions
• Carrying out pilot studies
• Building in flexibility from the start
• Taking precautionary action
• Developing less risky options, such as making
  less use of leading edge technology
• Reinstating, or developing different options
• Abandoning the project because it is too risky
• Transferring risk through contractual
  arrangements (insurance)

66
Risk treatments

• Risk avoidance decision to not become involved in
  a risk situation.
• Risk acceptance An (informed) decision to accept
  the likelihood and consequences of a particular
  risk.
• Risk transfer Shifting the responsibility or
  burden for loss to another party through
  legislation, contract, insurance or other means.
• Risk reduction A selective application of
  appropriate techniques and management principles
  to reduce either likelihood of an occurrence or
  its consequences or both (e.g. Business
  Continuity Plans, Emergency Response Plans, IT
  Failover etc.
• Risk retention Intentionally or unintentionally
  retaining the responsibility for loss or the
  financial burden of loss within the organisation.

67
Private Investment

• Factors Affecting the Level and Form of Private
  Investment in Regeneration
• Report to the Office of the Deputy Prime Minister
• University of Ulster
• December 2002

68
said

• The primary reason why the private sector invests
  is the expectation of achieving above average
  rates of return to offset added risk.
• The private sector will not invest in
  regeneration areas without substantial public
  sector commitment.
• Arising from data deficiencies, market signals
  are weak and confused thereby creating conditions
  of uncertainty that deters investors.
• The lack of information is important in
  perpetuating misconceptions of regeneration areas
  in terms of risk and levels of return.

69
and

• The under-pricing of regeneration markets by the
  private sector is a symptom of the information
  deficit. Likewise the risk premia that investors
  place on regeneration locations reflect stigma
  arising from site contamination and the
  perception of low environmental quality.
• The private sector stresses the rational
  behaviour underpinning decision making and the
  perceptions of risk within regeneration areas.
• The private sector places considerable emphasis
  upon the effects of crime and security
  considerations.
• Community involvement does not necessarily
  conflict with private sector investment
  interests, indeed benefits such as enhanced
  security can be channelled to positive effect.

70
Benchmarking of urban regeneration performance

• ESRC Paper R000239291
• There is a perception that investing in
  regeneration areas involves high costs and risk
  and low demand, leading to under-pricing of
  property assets.
• The alternative view is that regeneration sites
  can give, long-term, above average returns which
  off-set the extra risk involved. 
• Until now, there has been little supporting
  evidence specifically about returns on
  regeneration property. This makes the message
  confused and can adversely affect the way
  investors perceive markets and make decisions.

71
Return on Investment

• Since 1990, the rate of return on investments in
  quality property in areas of regeneration has
  been greater than the recognised, all-embracing
  national property benchmark.
• Examining 187 regeneration area properties in
  2001, The Eight City All Property Regeneration
  Index showed an annualised rate of return of 12.8
  per cent over 20 years, compared with 10.3 per
  cent for the overall, accepted industry index
  figure. This indicates that returns for
  investments in regeneration areas are greater
  than those across the market as a whole.     

72
Retail

• Retail property in regeneration areas, with a
  return of 15.5 per cent, out-performs both
  industrial (12.8 per cent) and office (10 per
  cent) property. This high figure reflects the
  fact that many major retail premises, including
  shopping centres and retail parks, have gone into
  designated renewal areas.
•    

73
Risk

• Investors' perceptions of higher risk in renewal
  areas may be misplaced. The analysis shows that
  risk is lower per unit of return than in the UK
  All Property Index. Significantly, retail
  property has the least risk per unit of return,
  again demonstrating the importance of this sector
  within regeneration areas.     

74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
Contingency or risk management?

• Contingency often missed, or misunderstood, in
  project applications
• Contingency actions are those that will be
  undertaken in the event of a risk being realised
  not what will prevent its realisation

78
(No Transcript)
79
Assessed

• Most contingencies described are not really
  contingency strategies. If ERDF is unsuccessful,
  what impact will it have? Will funds be
  required from elsewhere, or will infrastructure
  development be curtailed? Given a funding
  profile to 2006, what are the restrictive time
  constraints?
• Risk of low grant-take up will simply mean less
  budget spent. Risks of Enterprise Centre
  development potentially much more serious. This
  aspect of the project is inadequately described,
  and has no contingency description. As such,
  this aspect of the project must be considered a
  serious funding risk.

80
Critical Path Analysis
81
Critical Path diagram
82
PERT (Program Evaluation and Review Technique)

• PERT is a variation on Critical Path Analysis
  that takes a slightly more sceptical view of time
  estimates made for each project stage.  Use the
  formula below to calculate the time to use for
  each project stage
• shortest time 4 x likely time longest
  time---------------------------------------------
  --------------6
• This helps to bias time estimates away from the
  unrealistically short time-scales normally
  assumed.

83
Sensitivity analysis

• A new IT system costs 1million and is expected
  to yield staff savings of 150,000 per year over
  a period of 10 years. Discounting at 3.5 per cent
  the NPV of these costs and benefits is 247,000.
• Suppose the estimates of staff savings assumed
  that the IT system would replace 15 staff with an
  average cost per person of 10,000. A possible
  sensitivity test is as follows what if the IT
  system replaces only 10 staff? Staff savings
  would then fall to 100,000 per year and the NPV
  turns negative (minus 168,000).

84
Common reasons for failure

• Failure to break into manageable steps
• Evaluation of proposals driven by price rather
  than value for money
• Lack of understanding and contact with suppliers
• Insufficient joint working
• Source Office of Government Commerce

85
Sustaining projects

• Tapered funding
• Criterion rather than timetable
• Exit strategy from start
• Community involvement
• Tapered exit
• Foster homes

86
Evaluation

• Evaluation is the logical extension of appraisal
• Most projects fail at evaluation because of poor
  initial appraisal in establishing measurable
  outputs or failure to monitor

87
Why evaluate?

• In-programme to assess progress re-direct
  (formative)
• At or near end to capture learning (summative)
• Vital part of project life-cycle
• Because the funders say so

88
Project lifecycle
89
Yet more history
90
(No Transcript)
91
(No Transcript)
92
Common problems

• Contract not passed to project staff, so
• Monitoring not undertaken project -generated
  data not captured
• Outputs not related to aims
• External data not available or not budgeted to
  acquire
• Perceived inflexibility of funders

93
Common problems (2)

• Evaluation commissioned too late
• Perceived non-relevant data not captured
  stories lost
• Little cross-disciplinary involvement
• Partnership progress not recorded
• Behavioural change ignored as an outcome

94
Government view

• Evaluation needs to be thought about at the
  beginning of a project even before programme
  development and implementation. In the past, this
  has not been the case and evaluation has often
  been an after-thought, just before the money runs
  out! Partnerships have paid lip service to
  evaluation and written evaluation into the
  Delivery Plan but have not carefully considered
  its purpose or how it will be carried out.
• We agree.

95
Evaluation approaches

• External statistics top down
• Client-based Survey, Case studies, diaries
  bottom up

96
Monitoring types

• Absolute quantity
• Return on capital
• Floors/baseline change
• Disparity reduction
• Soft outcomes and distance travelled
• Star outcomes

97
(No Transcript)
98
(No Transcript)
99
People or place?

• Population stats, or tracking individuals?
• Assisting escape?
• Profiling

100
Process

• Developing an evaluation strategy being clear
  about the purposes of evaluation, the methods to
  be used and who has the capacity to do it
• Designing the evaluation and, if you decide it
  should be commissioned externally, writing the
  specification for a contract
• Choosing the evaluator through transparent and
  effective advertising, interview and selection
  processes
• Managing the evaluation making sure the
  evaluator does what the partnership wants, when
  it wants it
• Using the results getting feedback and reports
  that help the partnership know whether it is
  achieving the outcomes it hopes for

101
Prove it! methodology

• The participation of local people this helps to
  ensure that the indicators reflect impacts that
  local people think are important
• Focus on outcomes Outcomes (impacts) are harder
  to measure than outputs or inputs. All the Prove
  It! indicators connect to outcomes
• Core indicators local people and Groundwork
  staff, assisted by NEF, developed indicators.
  These focus on social capital
• e.g. neighbours around here look after each
  other

102
Technical issues

• Local Evaluation of Regeneration Partnerships a
  good practice guide
• Old, but good on
• Deadweight
• Displacement
• Double counting

103
How much?

• WHO 10
• Sure Start 3 - 5
• NRU 2 5
• Typical 1.5
• of project budget
• We declare an interest.

104
The end depends on the beginning

• There is a connection between the rigour of
  the initial problem diagnosis, the quality of the
  programme design, and the effectiveness and
  durability of results. This is hardly surprising
  before you can be confident about the right
  solution, you need to know you are tackling the
  right problem.
• JRF Creating sustainable neighbourhood and estate
  regeneration

105
Thanks

• We hope you feel better equipped to tackle the
  difficulties of project development
• Thank you for your attention and contributions

106
Evaluation Forms