PENNDOT

1
PENNDOTS LTAP EVALUATION ACTIVITIES

• Kim M. Ferroni, PennDOT
• LTAP Project Manager
• Barbara T. Harder, B. T. Harder, Inc.
• LTAP Evaluations

2
PennDOT LTAP

• PennDOT owner and project management
• Contracted operations -- Pennsylvania State
  Association of Township Supervisors since
  December 2005
• Sponsored during 2007
• Training workshops, 227
• Technical assistance sessions, 89 on-site
  (field), off-site 251

3
PennDOT LTAP Evaluation Activities

• Training Workshops
• Happy Sheet information evaluation of the
  experience (facility, time, materials,
  instructor)
• Self assessment of learning
• Initiating pre- and post-testing
• Follow-up survey

4
PennDOT LTAP Evaluation Activities

• Technical Assistance
• Self assessment of learning
• 6 month follow-up evaluation
• General benefits assessment

5
Training Workshops

• Report some basic statistics
• 3 workshops for illustration
• Asphalt Roads Common Maintenance Problems
• Bridge Maintenance
• Work Zone Traffic Control
• Most frequent workshop offerings during 2 month
  period of 2007 (representative sample about 60
  of total)

6
What Evaluation Sheets Tell Us
Workshop Self Evaluation Rated Average Learning Increase
Asphalt Roads Common Maintenance Problems 68
Bridge Maintenance 85
Work Zone Traffic Control 99

• Based on rating of three specific learning goals
  for each workshop
• 100 rating means doubled the knowledge

7
Self Evaluation of Learning

• Self Evaluation of Learning
• Highest assessed learning reported in Work Zone
  Traffic Control Workshops
• Understand requirements or WZTC in accordance
  with state and national guidelines (83)
• Able to develop traffic control plans using
  PennDOT publication (100)
• Understand recent changes in work zone rules
  (115)
• 100 rating means doubled the knowledge

8
Use of Training Information

• 95 of workshop attendees reported that what they
  learned would improve job performance
• 76 will use the information they learned within
  6 months of the workshop
• 9 use within 1 year, 13 unsure

9
Happy Ratings
Criteria Rating
Effective instructor 4.7
Training materials relevant, easy to use 4.5
Can use materials as reference 4.6
Room, sound, visuals appropriate 4.5
Recommend to others 4.7
10
Follow-up Training Workshop Surveys

• Initiating use of Survey Monkey to determine
  follow-up actions (distributed 6 months after
  training)
• Performance
• Implementation
• Learning
• Do not yet have sufficient data to analyze

11
Maintenance and Safety Technical Assistance

• Technical assistance
• Off-site telephone, email
• On-site
• Maintenance and safety topics
• Traffic signs
• Engineering and traffic studies
• Work zone traffic control
• Guiderails, posting and bonding
• Evaluation requested for every tech assist
  session 40 response rate

12
Maintenance Safety Tech Assists Types of
Evaluation Respondent Organizations June 06-Nov 07
County, 0, 0
13
Maintenance Safety Tech Assists
Frequency of Benefits Evaluations ReceivedJune
06-Nov 07
14
Maintenance Safety Tech AssistsUsefulness of
Information Evaluations Received June 06-Nov 07

• 80 of respondents anticipate using information
  provided in Tech Assists within 6 months and 91
  within one year
• AND
• 94 of Tech Assist information adequately
  addressed the issue of concern

15
Maintenance Safety Tech Assists Happy
Ratings June 06-Nov 07

• Use as exception flag for problems
• Provides opportunity for suggestions and other
  feedback

Element Quality Rating
Staff effective 3.88
Materials relevant easy to read 3.82
Time appropriate 3.83
Maximum rating 4
16
Maintenance Safety Tech AssistsSummarized
Comments Evaluations Received June 06-Nov 07

• The LTAP staff was (multiple respondents) quick
  to respond, answered all my questions, helped
  to educate me throughout several processes, very
  helpful, polite, and professional, excellent to
  work with, easy to talk with, an encyclopedia of
  traffic engineering and detailed issues
• The information provided was (multiple
  respondents) understandable, enabled us to
  use the correct material for our signs, gave us
  confidence in knowing the correct way, delivered
  quickly
• We learned a lot. Written info will come in
  handy in the future also. Thank you for
  providing this type of assistance.

17
Technical Assistance Follow-up

• Follow-up evaluation for on-site visits, 22.6
  response rate
• Increase Safety and Do Job Right Way are
  leader benefits after 6 months

18
Types of Benefits

• Quantitative Benefits
• 10 of 34 Benefits Evaluations provided some
  quantitative information (29)
• Candidates for more in-depth assessment
• Qualitative Benefits
• 32 of 34 Benefits Evaluations provided some type
  of qualitative information (94)

19
Example Quantitative Benefits

• Engineering Traffic Studies
• Saved 10,200 did not have to do a traffic
  study
• Traffic Signs Guiderails
• 30 hours saved/month 11.50 rate/hour
• Saved the cost of putting guiderails at the wrong
  places and the time to put up the correct signs

20
Example Qualitative Benefits

• Risk reduced for walking and motoring public.
  Liability risk reduced for Borough and school
  district. Safety increased significantly.
• The project was beneficial to establishing better
  working relationship with the township.
• Better or cleaner stone for chip roads. Trying
  to make our tar chip projects last longer.
• Avoid possible law suits made drivers more aware
  of upcoming intersection.

21
Questions and Issues for Discussion

• Training workshop pre- and post-testing what
  works, what doesnt?
• Literacy issues
• Voluntary or mandatory
• When used
• Learning assessment from supervisors
• What experiences have others had?
• Response rates
• Effort required
• Benefits measurements what programs are doing
  this and how?
• Who is providing the information?

22
For More Information

• Kim M. Ferroni PennDOT LTAP Project Manager,
  kferroni_at_state.pa.us
• Carol Kilko Contract Manager, PSATS,
  ckilko_at_psats.org
• Barbara T. Harder B. T. Harder, Inc. Contract
  Staff, Evaluations, btharder_at_verizon.net