RESEARCH AND EVALUATION OF UPSKILLING PROJECTS

1
UPSKILLING PARTNERSHIP PROGRAMME
RESEARCH AND EVALUATION OF UPSKILLING PROJECTS DR
JOHN BENSEMAN PRINCIPAL RESEARCHER
Presentation for Summer Institute 2009 June 25
27, 2009
2
GREETINGS FROM AOTEAROA NEW ZEALAND

• Bonjour
• Kia ora tatou (Maori)
• Malo lelei (Tongan)
• Talofa lava (Samoan)
• Kia orana (Cook Islands)
• Ni sa bula vanaka (Fijian)
• Namaste (Fiji Indian)
• Fakaalofa lahi atu (Niuean)
• Warm Pacific greetings!

DOCUMENT TITLE SUB TITLE
3
GREETINGS FROM AOTEAROA NEW ZEALAND

• New Zealand Canada
• Population 4.3 m (122nd) 33.6 m (36th)
• Sq kms 268,000 (75th) 2.6m (2nd)
• Total sheep 40m 1.1m
• Vote for women 1893 1918

DOCUMENT TITLE SUB TITLE
Shrek the Sheep
4
THE TEAM

• Dr John Benseman - Principal Researcher
• Dr Cathy Wright Senior Researcher
• Anne Alkema Research Manager
• Dr Earl Irving - Statistician
• Return on Investment (ROI) studies
• Dr Tim Maloney (University of Auckland)
• Dr Geoff Perry (Auckland University of
  Technology)

DOCUMENT TITLE SUB TITLE
5
THE (ORIGINAL) CONTEXT

• National surveys (IALS ALL)
• Low unemployment and skills shortages
• High rates of overtime
• Need to grow skills
• Long-term goal of increased productivity

DOCUMENT TITLE SUB TITLE
6
THE NEW CONTEXT

• New government since November 2008
• Economic crisis
• Government policy detail still forming in the
  light of current situation, but includes
• High emphasis on maintaining employment
• Skills training for the workforce

DOCUMENT TITLE SUB TITLE
7
UPSKILLING PROJECT AIMS

• Research and evaluation will build an evidence
  base
• what factors engage and motivate employers and
  employees
• how barriers to uptake and access can be shifted
• which approaches and models work best and are
  most cost effective
• the role of government and other organisations
• the impact on workplace practices and
  productivity measures

DOCUMENT TITLE SUB TITLE
8
WORKPLACE LLN - RESEARCH LITERATURE

• Very few good quality research studies on
  workplace LLN
• The few that do exist, are of limited value
• Gray, A. (2006). Upskilling through foundation
  skills - A literature review . Wellington
  Department of Labour (http//www.dol.govt.nz/publi
  cation-view.asp?ID242 )
• This review has highlighted both the dearth of
  reliable evaluations of LLN initiatives and the
  difficulty of undertaking such evaluations
• The literature will be updated with the final
  report

DOCUMENT TITLE SUB TITLE
9
THE PROCESS (1)

• UPO negotiates 15 LLN programmes
• Partnership agreement
• signed - DoL, company, LLN provider
• Funding from Govt.
• Needs analysis
• Course based on NA
• Evaluation negotiated

DOCUMENT TITLE SUB TITLE
10
THE PROCESS (2)

• Pre-course interviews and assessments
• Course proceeds (on-going contact and visits
  during courses)
• Post-course interviews and assessments
• Individual company reports
• Overall report on all 15 companies (late 2009)

DOCUMENT TITLE SUB TITLE
11
UPSKILLING PROJECTS

• Diversity of
• Programmes
• Learners
• Industries
• Company size
• Geographical
• location

• 15 companies
• 19 courses
• 11 providers
• 548 initial enrollees

DOCUMENT TITLE SUB TITLE
12
THE COMPANIES

• Road construction (3)
• Meat industry (2)
• Health
• Wool processing
• Cleaning
• Transport
• Aged care
• Tourism
• Re-cycling
• Seafood processing
• Food manufacturing
• Engineering

DOCUMENT TITLE SUB TITLE
13
LOCATIONS
14
THE COURSES

• The courses
• Content is contextualised on
• LNA of company
• Four are embedded courses
• 11, small groups, classes
• 24 100 hrs long (av. 47 hrs)

• Teaching
• 1-2 hours a week
• 2 days / break / 2 days
• 2 12 months duration
• Teachers providers
• Varied experience in terms of
• LLN workplace teaching

DOCUMENT TITLE SUB TITLE
15
DATA SOURCES

• Interviews (pre- and post-course)
• Course participants (60 minutes)
• Supervisors managers
• Providers
• Assessments
• Go! for writing and reading
• ACER for numeracy
• Supervisors for workplace practices
• Course documentation and records

DOCUMENT TITLE SUB TITLE
16
Return on Investment (ROI) Evaluation

• Key to understanding changes in company
  productivity - 7 ROIs planned
• Very few true ROIs done many err on the simple
  side of a complex process
• Centrality of understanding the counterfactual
  what would have happened if the course hadnt
  occurred
• Controlled experiments
• Before/after comparison
• Difference-in-difference (DID) estimators
• DID Estimator with varying intensities of
  treatment

DOCUMENT TITLE SUB TITLE
17
Return on Investment (ROI) Evaluation

• Pre-programme and post-programme comparisons on
• Absenteeism
• Staff retention
• Wastage
• Re-work
• Health and Safety
• What role intangibles?
• First ROI report will be available mid-2009,
  second end of 2009

DOCUMENT TITLE SUB TITLE
18
CURRENT STATUS OF PROJECT

• Nine of the 15 projects have finished and been
  reported
• Remaining six projects due to finish by mid-2009
• To date (results presented today)
• 494 enrollees interviewed before courses
• Of the 494, 265 have been interviewed/assessed
  post-course
• Anticipated at completion
• overall total of 330 pre- and post- course

DOCUMENT TITLE SUB TITLE
19
(No Transcript)
20
COMPANY DRIVERS
21
(No Transcript)
22
PRE-COURSE DATA 494 ENROLEES (15 projects)

• 66 male
• Average age 39 yrs
• Average yrs with co. - 5.3
• Current job - 3.5 yrs
• Most are current or
• potential supervisors

• 23 Maori, 37 Pasifika, 31
• NZ European, 9 Other
• 34 ESOL

DOCUMENT TITLE SUB TITLE
23
PRE-COURSE DATA 494 ENROLEES

• IALS/ALL levels
• 14 Level 3
• 45 Level 2
• 41 Level 1
• Within IALS Level 1
• 9 Entry Level 1
• 8 Entry Level 2
• 24 Entry Level 3

• Self-ratings (1-6 scale)
• Interest in training 5.1
• Schooling experience 3.9
• Qualifications
• Average yrs. high school - 3.6
• 51 - no school quals.
• 42 - no post-school education

DOCUMENT TITLE SUB TITLE
24
RATING OF COURSES TUTORS
Courses are rated positively (4.9 on 1-6 scale)
by participants, especially for being different
from school Tutors are rated more positively
(5.4 on 1-6 scale) by course participants
DOCUMENT TITLE SUB TITLE
25
PARTICIPANTS LIKED

• Teaching content and skills relevant to their
  work, interests and issues
• Empathetic tutors who treated them as adults,
  listened carefully
• and tailored their teaching to their needs
• Courses that challenged them without being
  daunting
• Flexibility to pursue relevant topics in
  teaching and
• the organisation of teaching sessions

DOCUMENT TITLE SUB TITLE
26
PARTICIPANTS DIDNT LIKE

• Being promised that they could pursue their
  individual learning interests
• and were not able to carry through on this
• Not being challenged (esp. those with higher
  level LLN skills)
• Being impeded by the demands of learners with
  higher needs
• Being unsure about the purpose of their courses

DOCUMENT TITLE SUB TITLE
27
IMPACT ON LEARNERS - CONFIDENCE
DOCUMENT TITLE SUB TITLE
28
IMPACT ON LLN - READING SKILLS

• 205 with both pre post-course assessments (42
  not re-tested)
• Average increase in scaled score 31.2
• 85.4 increased their scaled scores
• 1.9 stayed the same12.7 regressed
• Cohens d effect size - .66
• The average change in scores attributed to the
  programme measured
• in terms of standard deviations
• 35 went up an IALS/ALL level
• 46 went up an English level

DOCUMENT TITLE SUB TITLE
29
IMPACT ON LLN - WRITING SKILLS

• Average increase in scaled score 15.8
• Cohens d effect size - .31
• 64.0 increased their scaled scores
• 11.3 stayed the same
• 24.6 regressed

DOCUMENT TITLE SUB TITLE
30
IMPACT ON LLN - SELF-ASSESSMENTS
DOCUMENT TITLE SUB TITLE
31
IMPACT ON WORK - INTERVIEW FEEDBACK

• more complete and accurate form-filling
• increased confidence in work roles, more likely
  to take initiative
• less frustration with workmates supervisors
• improvements in specific LLN skills such as
  measuring
• small improvements in attitudes to work, job
  satisfaction company
• small decrease in their interest in training

DOCUMENT TITLE SUB TITLE
32
SUPERVISOR ASSESSMENTS
33
IMPACT ON WORK - WORKERS FEEDBACK

• Asked if the course had changed how they are
  doing their jobs
• 38 - a lot better
• 41 - a bit better
• 21 - no change
• Nearly half (49) said that they thought the
  course had changed how they think about their
  jobs
• I'm more interested. I feel better because I
  can do things better
• Before, I wasn't sure of a lot of things like
  barcodes, how to charge,
• how to deal with customer complaints
• Now I know what I'm doing fully - not just
  pretending!

DOCUMENT TITLE SUB TITLE
34
WIDER IMPACT

• Asked if the course had changed the way they
  relate to their family and friends
• 17 - a bit
• 15 - a lot
• Im talking to my brother-in-law Ive always
  found him arrogant, but I can talk to him now
• I feel confident and people are listening. We
  have family meetings once a month now
• 17 of those with school-age children report
  they are now
• helping them with their homework more
• Impact is more on relationships and use of LLN
  than social participation

DOCUMENT TITLE SUB TITLE
35
WITHDRAWALS ATTENDANCE

• Considerable variation among courses (56-91 -
  average 75)
• Withdrawals usually due to workers leaving
  company work demands
• Attendance helped by
• Courses in work time
• On-site locations
• Pro-active management
• Attendance better measure than withdrawals

DOCUMENT TITLE SUB TITLE
36
GENERATIONS 2, 3, 4

• Most have developed second generation programmes
• Most have adapted their programmes
• based on the evaluation reports
• Some are reviewing their providers
• Most are incorporating LLN into their
• core training plans
• Most are becoming more innovative
• in their plans

DOCUMENT TITLE SUB TITLE
37
EMERGING ISSUES

• Terminology to describe the courses
• Recruitment processes
• ESOL
• Low-level learners

DOCUMENT TITLE SUB TITLE
38
FROM WHAT WE HAVE SEEN SO FAR

• Workplace LLN programmes do recruit target groups
• Running successful programmes is challenging, but
  feasible
• Champions are needed at all levels of the
  company, incl. supervisors
• Participants improve their LLN skills and how
  they use these in their jobs
• Tutors are variable in their impact on learners
• Programmes have an impact on how many
  participants carry out their jobs
• Impacts are variable across programmes

DOCUMENT TITLE SUB TITLE
39
(No Transcript)
40
(No Transcript)
41
REPORTS TO DATE

• Benseman, J. (2007). Interim report to Works and
  Learning Wave on TeamWorks in Whangarei and
  Gisborne. Wellington, Department of Labour.
• Benseman, J., A. Alkema, et al. (2008).
  Evaluation of CMP Upskilling Partnership project.
  Wellington, Department of Labour.
• Benseman, J. and G. Denny (2007). Evaluation of
  Downer EDI Works Upskilling Partnership project
  1st report. Wellington, Department of Labour..
• Benseman, J. and G. Denny (2007). Interim report
  to Works and Learning Wave on TeamWorks.
  Wellington, Dept. of Labour.
• Benseman, J. and G. Denny (2008). Evaluation of
  Spotless Upskilling Partnership project 1st
  report. Wellington, Department of Labour.
• Benseman, J., G. Denny, et al. (2009). Evaluation
  of Canterbury Spinners Upskilling Partnership
  project. Wellington, Department of Labour.
• Benseman, J., G. Denny, et al. (2008). Evaluation
  of Downer EDI Works Upskilling Partnership
  project 2nd report. Wellington, Department of
  Labour .
• Benseman, J., G. Denny, et al. (2008). Evaluation
  of Fletcher Construction Upskilling Partnership
  project - 1st report. Wellington, Department of
  Labour.
• Benseman, J., G. Denny, et al. (2008). Evaluation
  of Fletcher Construction Upskilling Partnership
  project 2nd report. Wellington, Department of
  Labour.
• Benseman, J., G. Denny, et al. (2009). Evaluation
  of Spotless Upskilling Partnership project - 2nd
  report. Wellington, Department of Labour.
• Benseman, J. and E. Irving (2008). Evaluation of
  Downer EDI Works and ITS Upskilling Partnership
  project. Wellington, Department of Labour.
• Benseman, J. and E. Irving (2009). Evaluation of
  Downer EDI Works Way2Work course. Wellington,
  Department of Labour..
• Benseman, J. and E. Irving (2009). Evaluation of
  Sanfords Upskilling Partnership project.
  Wellington, Department of Labour.
• Wright, C., J. Benseman, et al. (2009).
  Evaluation of QRS Upskilling Partnership project.
  Wellington, Department of Labour.
• Wright, C., J. Benseman, et al. (2009).
  Evaluation of CDDHB Upskilling Partnership
  project. Wellington, Department of Labour.
• Wright, C., J. Benseman, et al. (2009).
  Evaluation of Affco Upskilling Partnership
  project. Wellington, Department of Labour.
• Wright, C., J. Benseman, et al. (2008).
  Evaluation of Air New Zealand Limited Upskilling
  Partnership project. Wellington, Department of
  Labour.

DOCUMENT TITLE SUB TITLE
42
LITERACY, LANGUAGE AND NUMERACY (LLN)

• 1996 first national survey as part of 17 OECD
  studies
• 2006 second, follow-up survey, also part of
  OECD
• fewer at Level 1
• but still 1.14m NZ adults at L1 L2
• Importance of these surveys
• LLN is an issue for New Zealand adults
• made LLN a mainstream issue, no longer fringe

DOCUMENT TITLE SUB TITLE
43
(No Transcript)
44
THE IMPORTANCE OF LLN IN NEW ZEALAND

• Included in many policy and planning documents
  for New Zealand adults
• Work-related - key to upskilling the workforce
  and productivity
• Pre-employment
• Corrections
• Tertiary education

DOCUMENT TITLE SUB TITLE