Effective Measures:

1
Effective Measures

• The Process of Creating a Community Based
  Evaluation and Research Support System for the
  Voluntary Sector To Monitor Long Term
  Community-Wide Outcomes

Social Planning Council of Ottawa 2003
2

• Presented by Dianne Urquhart,
• Social Planning Council of Ottawa
• 236-9300 ext. 303 dianneu_at_spcottawa.on.ca
• Research by Henry Reimer,
• independent evaluation consultant
• Funding for the development work of this project,
  including the research, has been generously
  provided by United Way of Ottawa and the
  Voluntary Sector Evaluation Research Project

3
Current Issues in the Non-Profit Sector

• Current funding strategies encourage increased
  accountability, partnerships, diversification of
  funding sources, improved efficiency and
  innovation
• Agencies face increased demands for program
  evaluation, with a strong and specific focus on
  outcomes measurement
• These emphases, despite good intentions, place
  additional demands on the resources of
  nonprofits, causing crises in some agencies and
  less organizational stability within the sector

4
Examples of Evaluation Issues

• Lack of resources for evaluation
• Funder needs (outcomes) and agency needs (e.g.,
  improving programs) may not be the same with
  respect to evaluation
• Agencies are being asked for evidence of impacts
  beyond their sphere of influence
• Simple quantitative measures such as units of
  service do not indicate intensity of benefits or
  the difficulties of the local context
• Multi-partner programs are difficult to evaluate
  fairly, esp. if a partner fails to implement its
  portion well
• Lack of trust between funders and nonprofit
  agencies
• Some models mya not be appropriate for specific
  population groups

5
Research and Evaluation Resources in Ottawa

• Centre for Voluntary Sector Research and
  Development (Carleton University)
  www.cvsrd.org/eng/index.html
• Voluntary Sector Evaluation Research Project
  www.vserp.ca
• Centre for Research on Community Services
  (University of Ottawa) www.crcs.uottawa.ca
• Canadian Evaluation Society -- National Capital
  Chapter www.evaluationcanada.ca
• Independent Consultants
• Networks among similar services
• The universities and colleges

6
Research and Evaluation Resources Contd

• Community Based Research Network of Ottawa
  (CBRNO)
• www.spcottawa.on.ca/cbrno
  Ottawas One Stop On Line Research and
  Evaluation Resource
• Lunch-time forums
• Symposium
• Student Placements
• new initiative Targetted learning sessions
• Social Planning Council of Ottawa
• www.spcottawa.on.ca
• Information Request Service
• Ottawa-Gatineau Data Consortium
• www.spcottawa.on.ca

7
Ottawa-Gatineau Social Data Consortium

• Initiative led by Canadian Council on Social
  Development and Statistics Canada
• Social Planning Council of Ottawa is the local
  lead
• Provides extensive social data for the voluntary
  sector
• Some Ottawa-wide data will be made available free
  on the Social Planning Council website
• Voluntary organizations can join for a fee and
  access data relevant for their community of
  concern, training to use the data and mutual
  support (discussion group)

8
Ottawa-Gatineau Social Data Consortium

• United Way of Ottawa has provided some financial
  support for this initiative in Ottawa
• Membership fees for voluntary sector
  organizations are
• 5,000 for patrons
• 2,000 per year for primary users (extensive
  data)
• Scaled contributions for targetted users (data
  only for their community of concern)
• 2,000 over three years for organizations with an
  annual budget of 1million or more
• 1,000 over three years for organizations with an
  annual budget of 500,000 to 999,999
• 500 over three years for organizations with an
  annual budget of 250,000 to 499,999
• negotiations for those with a smaller budget

9
What is needed?

• A community support structure to enable Ottawas
  voluntary sector health and social services to
  evaluate long term system-wide outcomes on a
  selection of major quality of life issue

• Effective Measures
  The Bank of Knowledge

10
What Have Other Communities Done to Evaluate Long
Term Community Outcomes?

• Have used a variety of social development
  approaches
• Outcomes-based management
• Social inclusion, social exlusion
• Population health
• Social capital and Civil society
• Asset-based community development
• Comprehensive Community Initiatives

• See www.spcottawa.on.ca/cbrno for a more detailed
  discussion (Powerpoint Approaches to Social
  Development)

11
Outcomes Based ManagementImplications for
Evaluation General

• The approach puts a particular type of evaluation
  (impact evaluation) at the very centre of the
  program planning process.
• Funding decisions are made on the basis of a
  programs current ability to demonstrate results
• Results are demonstrated by the pre-established
  evaluation plan.
• See Treasury Board Secretariat
• http//www.tbs-sct.gc.ca/eval/tools_outils/rmaf_cr
  gar_e.asp

12
O.B.M. Implications for Evaluation Pros

• Systematically collected impact data improves
  decision-making, decreases uncertainty
• Allows one to identify excellent programs.
  Program providers can showcase successes,
  government can show effective use of public funds
• Full integration of evaluation into the planning
  process makes evaluation easier and more
  accurate. For example, it allows for baseline
  data collection

13
O.B.M. Implications for Evaluation Cons

• Demands considerable organizational resources and
  may not meet the programs evaluation needs
• Works against innovation and organizational
  learning/improvement by disallowing intuitive and
  incremental problem-solving approaches. More
  generally, the status quo is rewarded, the
  experimental is penalized.
• It discourages change. It does not deal with
  change well
• Quantitative approaches simplify social reality
  in a way that can distort real impacts. Fair
  comparison of dissimilar programs in different
  local contexts is very difficult in practice
• Despite an emphasis on impacts, the approach
  demands quick results (funding decisions do not
  wait for long term results)
• Associated with a business model in which fellow
  agencies are seen as competitors. Works against
  a sense of sector-wide collaboration and mutual
  capacity building
• Reduces trust between funder and provider

14
Population Health Implications for Evaluation
General

• By highlighting social complexity it provides a
  reality check with regards to the limits of
  evaluation
• Shows the value of research and theory creation
  to illuminate complexity and reduce it to
  measurable components
• Other social issues can be approached in this way
  (by identifying determinants, etc.)
• Shows utility of global survey research and other
  universal data collection for macro-level
  evaluation
• See Health Canada
• http//www.hc-sc.gc.ca/hppb/phdd/index.html

15
P.H. Implications for Evaluation Pros

• Encourages research
• Holistic acknowledges the complex nature of
  factors leading to health/ill health.
• Allows for evidence-based decision-making, i.e.,
  it is pragmatic.
• Is democratic in its focus on the entire
  population. This allows the use of survey
  research to evaluate results (in part).
• Puts the onus of theory development and
  evaluation development (such as identification of
  appropriate indicators) on funders rather than
  program providers

16
P.H. Implications for Evaluation Cons

• It is focussed on health issues. Indicators and
  factors for non-health issues are not defined
• Surveys of large populations eliminate
  consideration of local variation solutions
  generated are also macro-level and may not fit
  local conditions
• Large multi-site programs are difficult to
  evaluate due to local variation, even if a
  standardized program is used
• It inherits the weaknesses of the rational
  planning model. It is top-down, requires
  complete knowledge prior to action, and does not
  deal well with change

17
Social Exclusion Implications for Evaluation
General

• Much practical experience has accumulated due to
  wholesale adoption of this approach in the EU and
  the UK
• Tends to result in large complex programs
• A large number of new indicators have been
  developed to measure non-economic aspects of
  social exclusion
• A relatively new concept indicators continue to
  require research
• See Closing the Distance
• http//www.closingthedistance.ca/index.jsp
• See Social Exclusion Unit (UK)
• http//www.socialexclusionunit.gov.uk/

18
S.E. Implications for Evaluation Pros

• Considerable research, implementation and
  evaluation has been done
• The concept strongly relates individuals to the
  community in which they live
• Reintroduces a holistic approach to social
  development. Allows complexity of social reality
  to shape its boundaries
• Allows visions of a good society to enter
  political debate
• The focus of the approach on global societal
  improvements allows the (partial) use of large
  scale evaluation methods such as social surveys.
• Fits well with Canadian social ideals

19
S.E. Implications for Evaluation Cons

• Talk about achieving a good society sounds
  expensive
• Inter-sectoral projects will meet resistance in
  government
• Local indicators to measure progress in
  inclusion would have to be developed
• Local funders need to buy into the indicators,
  some of which would be experimental
• Complex programs result in complex evaluations,
  demanding more evaluation capacity

20
Social Capital / Civil SocietyImplications for
Evaluation General

• The approaches provide strong incentive to view a
  healthy heterogeneous non-profit sector as a goal
  in itself
• Both social networks and organizational networks
  provide benefits that cannot be accessed by any
  other means.
• Evaluations of networks -- social networks
  (social capital) or organizational networks
  (civil society) -- raise new issues, require new
  vision, new indicators. There is room here for
  some exciting research!
• See Royal Roads University Victoria B.C.
• http//e-dialogues.royalroads.ca/scdialogue/scinfo
  .htm
• World Bank
• http//www.worldbank.org/poverty/scapital/index.ht
  m

21
S.C. Implications for Evaluation Pros

• The social capital approach highlights the
  tremendous value of trust in creating productive
  networks
• The approaches force one to look beyond
  individual organizations (or persons) to examine
  how they fit into social structures that surround
  them.
• Civil society is the domain of true participatory
  devt. It looks at the distribution of power
  in society and reconsiders the values of
  community organizing.
• In Canada, where it is sometimes difficult to
  separate nonprofits from government, these
  approaches make the distinction clear, and
  highlight the specific pros and cons of the
  nonprofit sector as a distinct entity.

22
S.C. Implications for Evaluation Cons

• They are relatively new concepts and need to be
  more clearly defined. E.g., what types of
  organizations, how many, what relationships, make
  a "healthy civil society?
• Funders tend to fund organizations that are
  familiar, or similar to themselves and this is a
  difficult barrier to cross without the
  appropriate academic argument.
• In the case of civil society, the implicitly
  political nature of the debate can make funders
  uneasy
• The concepts can be bent to accomplish goals of
  different political agendas. E.g., to reduce
  social exclusion or to unload govt
  responsibilities onto the nonprofit sector
• "Capacity building" efforts are not currently in
  style. Funders tend to like short-term projects
  that target the disadvantaged rather than service
  providers

23
Asset Based Community DevelopmentImplications
for Evaluation Pros

• Highly participatory local groups determine
  priorities, develop projects based on their
  assets, and evaluate their actions. It
  highlights the inappropriateness of funder-led
  development
• Participatory programs are always more likely to
  accept evaluation as a valuable process, to
  generate valid evaluation results, and to use the
  evaluation results directly to improve efforts
• It avoids a scenario in which community leaders
  are forced to insult their own communities to
  gain funding (by highlighting only deficiencies)
• Ottawa communities have a lot of assets
• See The Asset-Based Community Development
  Institute
• http//www.northwestern.edu/ipr/abcd.html

24
A.B.C.D. Implications for Evaluation Cons

• May not meet funders needs. Incremental
  planning is preferred bureaucratic reporting is
  de-emphasized
• Success is not assured assets (including
  leadership and volunteerism) vary greatly from
  one community to the next
• Funder/community relationships require more
  negotiation. Trust on both sides is necessary
• Quantitative outcome evaluation can be much more
  difficult in a fluid human (rather than
  laboratory-like) environment

25
Comprehensive Community Initiatives Implications
for Evaluation General

• Work across functional areas (social services,
  health care, schools, economic and physical
  redevelopment) in an effort to launch a
  comprehensive attack on social and economic
  constraints that lock poor children and families
  in poverty.
• Recognizes that the science of community
  research, especially with respect to
  comprehensive and complex initiatives, is
  under-developed
• Approach is developing new measurement tools and
  methods
• See Aspen Institute
• http//www.aspeninstitute.org/Programt1.asp?bid12
  64

26
An Action Plan

• SPC envisions a process in which funders and
  non-profit organizations, within each of several
  topic areas (e.g., housing and homelessness,
  inclusion for people with disabilities), discuss
  their organizational information needs and come
  to consensus about the following
• What truly matters in this topic area? The
  discussion should take equal note of agency and
  funder visions for social development.
• What combination of indicators, evaluation
  methods, and social development approaches will
  adequately capture changes in these vital
  matters, while meeting both agency and funder /
  government information needs?
• What research is necessary (at all levels) and
  how can it be made available?

27
An action plan

• These topical committees will be coordinated by
• -a broad based committee of nonprofits and
  funders to plan and coordinate the development of
  the model and advocate for the model among
  non-participants
• -a subcommittee to oversee technical development
  of internet tools
• - a subcommittee to advise on project content,
  including continuing development of an
  overarching vision for social well-being in
  Ottawa and an appropriate theory of action for
  getting there
• -The various topical committees will continue to
  revise the topical frameworks as experience is
  gained, data is analyzed, and academic research
  informs about the relationships between factors.
  

28
Our Development Process Incorporates

• Building Trust Active dialogue between funders
  and non-profits to build a mutually acceptable
  evaluation framework
• Planning for Sustainability Within the
  initiative but also within the sector

29
Our Development Process Incorporates

• Bridging the Gap between the desire and the
  capacity of individual voluntary sector
  organizations to undertake and use meaningful
  research and evaluation
• Greater collaboration between academic
  institutions and community based organizations
• An increased role for funders in research and
  evaluation
• An increased role for nonprofit administrators in
  research and evaluation
• Research and evaluation related to the capacity,
  short and long term needs, and sphere of
  influence of the individual organizations,
  funders and government departments
• Building the capacity of sector partners /
  developing a learning community

30
(No Transcript)
31
Would Involve

• Advocacy for mixed methods in evaluation
• Some degree of standardized data collection
• Clusters of indicators and methods in different
  areas of social concern
• Ongoing research into appropriateness and
  validity of indicators, methods and best
  practices in different topic areas
• Resource and capacity development
• Integration with the Ottawa-Gatineau Data
  Consortium, including a website which would
  likely include GIS mapping and other interactive
  tools

32
Assumptions

• Funders and governments will continue to need
  outcomes data
• Nonprofits can benefit greatly from appropriate
  evaluation information, including outcome data
• Funders and nonprofits are willing to work
  together on evaluation in a participatory way
• The effort spent in building an evaluation
  framework will be recouped by a subsequent
  decrease in the demands placed on individual
  program managers and by better information for
  funders
• While holistic approaches are necessarily more
  complex, simple is better with regards to
  evaluation activities. The goal is to develop
  the simplest evaluation tools possible that allow
  real appreciation of impacts
• There will not be a one-size fits all evaluation
  model
• Funders, government and non-profits all want to
  know, Are we making a difference?

33
Next steps

• Establish partnerships in the Ottawa-Gatineau
  Data Consortium
• Establish stage one of the Bank of Knowledge with
  foundational community accounts
• Identify key participants in the Advisory Group
  (Participants meet once a month)
• Expand Ottawas One Stop On Line Research
  Resource
• Begin targetted learning sessions on requested
  evaluation and research challenges
• Consult with the sector through a variety of
  means on the form and content of Effective
  Measures and the Bank of Knowledge
• Put the various components in place (including
  technical advisory committee, funding, etc.)
• Pilot Effective Measures The Ottawa Bank of
  Knowledge

34
Your Involvement

• Make arrangements to join the Ottawa-Gatineau
  Data Consortium
• Contribute your existing completed Ottawa based
  research to the Bank of Knowledge (form in your
  package)
• Fill in the form with respect to targetted
  learning sessions. If you have expertise to
  contribute to such sessions, identify yourself as
  a potential volunteer
• Give us feedback on foundational community
  accounts
• Identify key participants in the Advisory Group
  and volunteer to join if you feel you can
  contribute
• Participate in upcome consultation sessions
• Contact Dianne Urquhart if you have suggestions
  or would like to contribute in some other way to
  the development of Effective Measures The Bank
  of Knowledge (236-9300 ext. 303
  dianneu_at_spcottawa.on.ca)