Next Steps in Skill Assessment

1
Next Steps in Skill Assessment

• November 30, 2004
• Webcast

John C. Foster Stanley Rabinowitz Mary Shumway
2
The Afternoon OverviewWhat Are We Going to See?

• Policy Perspective
• People Perspective
• Pennsylvania Perspective

Or, how many more words we can think of that
begin with the letter P
3
Policy Perspective
What can we surmise from reading the signs?
4
Policy Perspective

• Accountability system evolving (CAR)
• Data quality varies
• Nationally data not used for decision making
• Perkins IV
• Both versions include language on technical
  assessment as an allowable expense
• Language enhancing pathways and clusters

5
CAREERCLUSTERFRAMEWORK

6
Knowledge and Skills Areas

• Academic Foundations
• Communications
• Problem Solving and Critical Thinking
• Information Technology
• Organizational Systems
• Safety, Health and Environmental
• Leadership and Teamwork
• Professional Ethics and Legal Responsibilities
• Employability and Career Development
• Technical Foundations

Can Assessment be far behind?
7
Policy Perspective

• An industry cluster study
• NGA embraces targeted industry clusters
• PA BCC Study

Emphasizes the need for technical training and
assessment!
8
People Perspective
What does everyone want?
9
People Perspective

• Employers need assessment
• To certify that employees have occupational
  competence (not just soft skills) Need Proof?
  Try the Sunday morning research project
• Employees need assessment
• To help them climb their career ladder
• To seize opportunities or advancement
• Customers need assessment
• To assure of quality of product or service
• Remember what ASE did for auto dealership service

10
Pennsylvania Perspective
What do things look like in the Keystone state?
11
SecondaryCareer Technical EducationDelivery
System
81 Area Career Technical Schools 33 School
Districts (8)
Erie
Susquehanna
Warren
McKean
Potter
Tioga
Bradford
Crawford
Wayne
Lacka- wanna
Wyoming
Forest
Elk
Venango
Sullivan
Mercer
Cameron
Pike
Lycoming
Clinton
Luzerne
Jefferson
Monroe
Butler
Clarion
Columbia
Lawrence
Clearfield
Montour
Centre
Union
Armstrong
Carbon
North- umberland
Indiana
Northampton
Beaver
Snyder
Mifflin
Schuylkill
Lehigh
Cambria
Juniata
Blair
Dauphin
Allegheny
Berks
Bucks
Perry
Lebanon
Westmoreland
Mont- gomery
Huntingdon
Washington
Cumberland
Lancaster
Chester
Philadelphia
Fayette
Somerset
Bedford
Fulton
Franklin
Adams
Greene
York
Delaware
12

• The Real Challenge
• The Technical Academic Rigor of the Workplace
  is Increasing Dramatically for ALL groups

S0.How do we assess it?
13
Policy Standards

• State Law Chapter Four
• Required Academic and Technical Assessment
• Assessment Plan
• Grades 3, 5, 8, and 11
• Reading, Writing, and Math
• 8 other sets of standard (not currently assessed)
• Technical Assessment upon program completion

14
Industry Standards

• Recognition and Encouragement to participate in
  over 100 programs
• Criteria for Recognition
• Nationally validated, vendor neutral curriculum
• Teacher training
• Facility certification

15
Industry Assessment
LPN
NA
16
NIMS Credentialing as Program Performance
Measures
Pennsylvania Example

• 2003-2004 Mandatory Program began in March
• Results as of 6/12/04
• 15 Advanced Level 4 or gt credentials
• 20 Proficient or Skill Certificate Level 3
  credentials
• 20 Basic Level 2 credentials
• 26 One Credential
• 216 Total Credential Exams Attempted
• 196 Total Credentials Earned
• 91 Success Rate

17
Secondary Industry AlignmentEarly Results
Capitalize on Certifications
PDE CATS Data 2002-03
18
Next Steps in Skill Assessment

• November 30, 2004
• Webcast

John C. Foster Stanley Rabinowitz Mary Shumway
19
Utahs Skill Certificate Program
Mary M. Shumway, State CTE Director Utah State
Office of Education
20
Utah

• School Age Population
• Average Family Size
• Lowest Per Student Expenditure in US
• 2,200
• High Tax Effort

21
Utah

• 40 School Districts
• 110 High Schools
• 10 Post-Secondary Institutions

22
Utah
State Graduation Requirements 1 CTE Credit
Required .5 Computer Technology Required
23
Utah

• 36,000 students per grade
• Grades 9-12
• 140,000 students in CTE
• --non duplicated count
• 186,000 students in CTE
• --duplicated count

24
History
1986 -Began some skill testing
1995 -Request for additional CTE
funding - Utah Legislature Increased
funding Tied to student performance
25
Purpose

• Verify Skills
• Improve Instruction
• Alignment
• Post Secondary Education
• Business and Industry

26
Funding

• No Funding for Implementation
• Utilize some Perkins funds
• Assess districts for test administration
• Incentive Funding for Districts
• Reimbursed based on performance
• 50 - 200 per certification

27
Students Tested
1995-96 32,638 1996-97 52, 259 1997-98
77,011 1998-99 97,529 1999-00
114,417 2000-01 131,173 2001-02 145,262 2002-0
3 159,036 2003-04 170,000
28
Skill Certificate
State Committee With Oversight Local District
CTE Directors State Staff Testing
Experts State Staff
29
Skill Certificate
Utilize 25 National/State Exams Developed
133 State Tests
30
Industry Exams
Sample of Industry Exams Utilized NATEF Cert
ified Network Assistant ASE Microsoft Cosme
tology A Certified Nurse Assistant Oracle
(OCP) Emergency Medical Technician Cisco
(CCNA) Pharmacy Tech Network Pro
Start Certified Internet Webmaster Linux
INet
31
State Exams
Sample of State Exams Developed Drafting Accoun
ting Medical Assisting Cabinetmaking Business
Management Dental Assisting Auto
Collision/Repair Banking Finance Biotechnology
Automotive Service Business Law Medical
Anatomy Welding Desktop Publishing E-Commerce
Marketing Machine Tool Computer
Technology Economics Graphic Arts Spreadsheets
Database Entrepreneurship TV
Broadcasting Web Page Design Foundations of
Technology Electronics Word Processing Principl
es of Engineering Carpentry Adult
Roles Agricultural Systems Technology Plumbing
Child Development and Care Animal Science
Technology Electrician Interior
Design Floriculture Greenhouse
Management Commercial Art Food and
Nutrition Plant Soil Science Commercial
Photo Culinary Arts Agricultural Business
Management Law Enforcement Textiles and
Clothing Natural Resource Management Into to
Information Tech Marketing Agricultural
Science Tech Multi-media Fashion
Merchandising Exercise Science/Sports
Medicine Retailing Travel Tourism Engineering
Design Development
32
Test Development
Establish a core set of skills/standards -Identif
ied by national/state skill standards -Input of
business and industry -Teacher
involvement -Standard and objective format
End of course or end of program -Semester, year
long, or end of program
33
Test Development
Assessments Developed -Test items developed
Pilot tests Pilot questions to develop test
bank -Performance assessments developed -Test
items analyzed Use data to revise (point
biserial) Use testing expert teams
34
Components of Test
Written Component 80 questions 10 of which are
pilot questions Performance Component Teacher
evaluation Summary record of student
completion Marked on question 81
35
Testing Procedures
Test coordinator at each school Order tests Test
booklets/scan sheet sent to school Teacher
administers tests Tests returned to state office
and scored Certificates issued back to
schools Reports to teachers, schools, district
36
Testing Procedures
24 Hour Turn Around Time Certification Rate - 80
Pass written exam at 80 Pass performance
objectives 80 Students Receive
Certificate Name and course listed on
front Competencies listed on back
37
Testing Procedures
Reports Teacher Standards for each
student Summary of standards School, District,
State Use State NCLB Terminology Substantial
80 and above Sufficient - 65 to 79
38
Reports
39
Reports
40
Successes
Alignment All Districts Participate Teacher
Involvement Articulation -Dual Enrollment -Tech
Prep Scholarships/Awards
41
Successes
Recognition of the Legislature Recognition of
Business Industry Stature and Visibility of
CTE Perkins Accountability Staff Development High
participation District support
42
Web Site
www.usoe.k12.ut.us/ate/Skills/skills.htm
43
Web Site
Click program area on the right side of the page.
Then tests offered will display.
44
Web Site
Click on the test to get protocol page.
45
Questions
46
Challenges
-Initial implementation fast track -Small
sample size with some tests -Limited resources
Test development, validation, refinement Cost
of national/industry exams
47
Future of Skill Testing
What is role of skills testing with NCLB,
academic achievement, school improvement,
state initiatives, etc.?
-Terminology -Rigor and Accountability -
School Performance drop out rate state
reporting/growth model graduation requirement
with area of focus
48
Future of Skill Testing
What is the impact on career clusters and post-
secondary transitions? -Utah Career
Clusters Comprehensive Guidance Involveme
nt on national career cluster teams Tech
Prep -Dual Enrollment/Articulation 74,000
credit hours in CTE annually Utah College of
Applied Technology
49
Future of Skill Testing
How will the program fit into national skill
verification initiatives? -Career Clusters
Knowledge and Skill Sets? -Future Perkins
Legislation? -Nationally Recognized Exams?
50
Continue to
-Improve technical quality -Provide staff
development -Develop articulation with
post-secondary -Involve business and
industry -Increase awareness -Utilize national
exams where possible -Move to on-line testing
where possible -Integrate into career clusters
51
In Summary

• Verify Skills
• Improve Instruction
• Alignment
• Post Secondary Education
• Business and Industry

52
For More Information
USOE.k12.ut.us.us/ate/skills/skills m.shumway_at_sc
hools.utah.gov
53
Utahs Skill Certificate Program
Mary M. Shumway, State CTE Director Utah State
Office of Education