Evaluation

1
Chapter 16

• Evaluation

2
Role of Evaluation

• Evaluation marks the end of the event,
  highlighting the need to look back and review all
  aspects of the operations. It is also the
  beginning of the planning stages for the next
  event, requiring a look forward aided by new
  information
• The goal of evaluation is to gather information
  that may then be used to make decisions regarding
  the planning and delivery of the event in the
  future

3
Role of Evaluation

• A well-designed evaluation strategy
• Identifies critical areas of performance
• Supports the development of realistic standards
  that identify success
• Implements evaluation tools and techniques that
  effectively compare reported results to these
  standards

4
Why Evaluate?

• Reflect on past
• Plan for future
• Gather and retain data
• Support the decision-making process
• Build a sense of continuity
• Strategic planning requires strategic evaluation

5
Getting Started

• Who will use the information?
• What is it that you want to be able to decide
  based on the evaluation?
• What information is needed?
• How is the information to be collected?
• How will the information be analyzed?
• When will the results be needed?

6
What to Evaluate

• Critical Areas of Performance
• Administration
• Operations
• Marketing
• Risk management
• Standards and Expectations SWOT
• Operational Effectiveness
• Customer Service
• Marketing Strategy
• Economic Impact

7
I. Administration Evaluation

• Resource Management
• Material Resources
• Time
• Information
• Technology

• Resource Management
• Fiscal Resources
• Budgeting
• Purchasing
• Sponsorships
• Revenue Sources
• Human Resources
• Committee
• Staff
• Volunteers

8
II. Operations Evaluation

• Stakeholder Experience
• Site Management
• Infrastructure
• Logistics
• Communications
• Program Effectiveness
• Technical Aspects

9
III. Marketing Evaluation

• Marketing Plan Management
• Public Relations
• Promotions
• Product Management
• Price
• Position

10
IV. Risk Management Evaluation

• Risk Assessment
• Compliance
• Emergency Management
• Health and Safety
• Legal and Ethical Considerations
• Security
• Insurance

11
Methods of Evaluation

• Interviews
• Surveys and Questionnaires
• Documentation Review
• Focus Groups
• Direct Observation
• Case Studies
• Pre and Post Tests
• Logs
• Rating sheets

12
Problems with Evaluation

• Inappropriate instruments
• Reliability
• Statistical errors
• Incomplete data
• Ignoring negative results
• Confidentiality
• Memory decay in post event surveys
• Language barriers

13
Problems with Evaluation

• Barriers to Customer Participation
• Length of surveys
• Location and timing of surveys
• Training and effectiveness of interviewers
• Multi-entrance/exit venues

14
What Research Tells Us

• Standardized models for event evaluation
• Pre-event areas for evaluation
• Potential risk exposure for the organizing agency
• Probability of success
• Level of financial support needed
• Event managers capabilities
• Carlsen, Getz Soutar (2001)

15
Pre-Event Evaluation Model

• Potential number of visitors
• Potential economic impact
• Compatibility, Capacity and Suitability of venues
• Time of year of the event
• Growth potential of the event

16
Pre-Event Evaluation Model

• Expected level of local support
• Potential regional benefits
• Enhanced prestige for the local community
• Potential community benefits
• Benefits to host organization
• Potential environmental impacts
• Potential employment impacts
• Fit with destination

17
Pre-Event Evaluation

• Image enhancement
• Potential for sponsorship
• Potential for media impact
• Catalyst for infrastructure development
• Potential for links with other events
• Regular staging in present location

18
Post-Event Evaluation Model

• Economic impact at the local level
• Number of interstate and international visitors
• Direct visitor expenditures
• Financial results profit/loss
• Problem free operations
• Sponsor satisfaction
• Total attendance

19
Post-Event Evaluation

• Overnight visitors in area
• Value of media coverage
• Positive community attitudes
• Employment creation
• Cost-benefit analysis
• Environmental impacts
• Community socio-cultural impacts

20
Post-Event Evaluation

• Yields per visitor
• Infrastructure improvements
• Urban renewal
• Prestige Image enhancement
• Enhanced potential to host future events
• Higher volunteerism
• Event expertise
• Future use of purpose built facilities

21
Analyze the Results

• Strategic Results
• Quantitative
• Raw Data
• Measures of Central Tendency
• Causation and Correlations

22
Analyze the Results

• Qualitative
• Focus Groups
• Narrative Interviews
• Moods, Feelings, Impressions
• First-Person Accounts
• Tabulation of Qualitative Information

23
Implement the Results

• Strategic Implementation
• Apply results to future SWOT analysis
• Timing of implementation is critical
• Determine the audience for dissemination of
  results

24
The Evaluation Document

• Title Page
• Table of Contents
• Executive Summary
• Purpose of the Report
• Background of the Event
• Organizational description and history
• Description of the event
• Goals of the event
• Outcomes and performance measures
• Activities presented
• Service delivery details
• Staffing

25
The Evaluation Document

• Evaluation goals
• Methodologies
• Type of data collected
• Data-collection procedures
• Data-analysis procedures
• Limitations of data
• Interpretations and Conclusions supported by data
• Recommendations
• Appendices
• Instruments used
• Tabulated results
• Narrative statements regarding evaluation process
• Relevant case studies
• Significant literature

26
Why Festivals Fail

• Failure is Relative
• Shutting down an event
• Planned but never delivered
• Serious problems at the event
• Customer dissatisfaction
• Accidents and injuries and risks
• Weather-related interruptions
• Change in focus, name or venue of the event
• 
  

27
Why Festivals Fail

• Poor weather
• Disgruntled merchants
• Lack of volunteers
• Lack of vision and management
• Lack of future funding
• Banning of alcohol sales in public parks
• Incompetent event managers
• Fiscal mismanagement
• Lack of adequate sponsorships
• Riots

28
Why Festivals Fail

• Ranked reasons for Failure
• Internal factors
• inadequate marketing or promotion
• lack of advanced or strategic planning
• inattention to program quality or service
• External factors
• weather
• competition from other events during the same
  time period