How to Evaluate

1

How to Evaluate Driver Education Programs
2
Published August 2006 Prepared by Northport
Associates Co-funded by BMW of North
America Available online at www.aaafoundation.o
rg
3
The Problem

• Traffic crashes remain the 1 killer of teens
• Most research indicates that driver education
  courses are not effective tools to reduce these
  deaths and injuries

4
Purpose of This Research

• To develop comprehensive guidelines on how to
  evaluate the effectiveness of novice driver
  training and education programs

5

• Does novice driver training and education make an
  impact on skills and safety?

6
History of Evaluation in Driver Education

• Long history Jim Nichols NTSB paper
• Recent reviews little critical analysis
• One recent meta-analysis
• Mainly summative evaluation
• Experimental RCTs records, surveys
• Quasi-experiments /- statistical control
• Ecological studies population effects

7
History of Evaluation in Driver Education

• Prominent Findings
• DeKalb
• Reviews
• Mayhew Simpson
• Australian reviews
• Systematic reviews
• Elvik Vaa meta-analysis

8
Management Overview
Program Evaluation Definition
the systematic assessment of the operation
and/or outcomes of a program or policy, compared
to a set of explicit or implicit standards as a
means of contributing to the improvement of the
program or policy. Weiss, 1998
9
Why Evaluate Driver Education Programs?

• 1. We have big questions left unanswered
• Do programs have logical learning objectives?
• Do they achieve their learning objectives?
• Do they have the desired safety impacts?
• Do some programs lead to better results than
  others?
• Which components of programs contribute to
  success?
• How can programs be improved to maximize impact

10
Why Is Systematic Evaluation Important?

• 2. Credible answers allow us to
• Identify program strengths weaknesses
• Measure progress objectively
• Identify ways to improve programs
• Share objective information about what works
• Influence policy funding
• Build an effective base of knowledge

11
What Is Important to Know About Evaluation?

• Key Concepts
• 1. Formative Evaluation
• Intended to improve program logic, content,
  delivery
• Asks
• Does the program operate according to plan?
• Is it meeting its teaching objectives?
• How can it be improved?

12
What Is Important to Know About Evaluation?

• Key Concepts
• 2. Summative Evaluation
• Intended to prove or sum-up results benefits
  
• Asks
• Does the program make students better drivers?
• Does it reduce crashes?

13

• Evaluating Driver Education Evaluation
• Has evaluation improved proved driver
  education?
• Negative influence on beliefs and policy
• Weak in formative/process areas
• Statistical weaknesses in summative evaluations
• Unsystematic hit run research
• Lots of room for improvement

• Knowledge know-how

• Continuous improvement

• Progressive policy

• Positive, cost-effective program impacts

14
Five Steps to Evaluating

15
Overcoming Barriers to Evaluating?

• What if researchers find your program does not
  work?
• Systematic program evaluation is the key
• Ongoing, stepwise system of evaluation
• Small steps continuous improvement
• There should be little risk of nasty surprises
• Still, plan for unexpected findings

16
Retooling Driver Education for Progress

• World has changed around driver education
  reform is required

• Most jurisdictions lack resources political
  will

• Improve Uniformity there cannot be 60 best
  ways

• Behavior change is tough programs need to
  reflect this

• Evidence-based development is badly needed

• RD comprehensive evaluation need high
  priority

17
The Future Bridging the Great Divide

• Practitioners managers believe in driver
  education

• Researchers typically believe driver education
  doesnt work

For more effective driver education in the
future, key people in both camps must work
together toward improving evaluation and using
evaluation tools to improve driver education.
Management Overview p.15
18
What is next?
Guidelines
Evaluating Driver Education Programs Comprehensiv
e Guidelines Management Overview How-To Manual
Implementation of Guidelines
Large-Scale Evaluations of Driver Education
Programs
19
The AAA Foundation for Traffic Safety is a
501(c)(3) public charity located in Washington,
DC that is dedicated to saving lives and reducing
injuries. It is supported by donations from
AAA/CAA Clubs, AAA/CAA members, and other
organizations associated with AAA/CAA.