Technical Skill Assessments Overarching Issues

1
Technical Skill AssessmentsOverarching Issues

• Common Concerns
• Intended State Approaches
• Critical Definitions
• Data Quality Institute
• Savannah, Georgia
• May 19, 2007

2
Overarching Issues Topics

• Don Hilber, DTI
• Technical Skills Assessment Study Group advice
  for DQI agenda / associated research
• Results of Questions for States issues of
  greatest importance stage of development
  intentions in transition plan beyond
• Framework for approaching tech skills indicators
• Bob Sheets, U of IL
• Recap recommendations from 2005-2006 DQI
• Perkins IV realities what still applies and
  what does not
• Facilitated discussion Q A

3
Technical Skill Assessment Study Group

• Spun off of Next Steps Working Group
• 25 states, OVAE, associations and consultants
  about 50 get notices
• Accountability / data experts and curriculum /
  assessment expert some administrators
• Eleven calls starting Jan 10, 2007
• Invited private companies to address group and
  take questions
• Informed DQI desired other research knowledge
  of others, (deferred) inventory

4
Study Group Most Active States
Alaska Secondary North Carolina Secondary
Arizona Secondary Ohio Both
Arkansas Both Oklahoma Secondary
Florida Secondary Oregon Secondary
Hawaii Secondary South Carolina Secondary
Indiana Postsecondary Tennessee Postsecondary
Iowa Secondary Texas Both
Michigan Secondary Utah Both
Minnesota Postsecondary Virginia Secondary
New York Postsecondary Wisconsin Both
5
Experts Invited to Address Technical Skill
Attainment Study Group

• NOCTI (Foster )
• ACT (Klemme )
• VTECS (McCage, Hattaway)
• XPAND Corp (Osman, Butler)
• OSU (Mahlman )
• ABG / Skills USA (McClelland )
• Others Post-DQI ?

6
Questions Asked of States

• Level of Experience with Technical Skill
  Assessments Secondary and Postsecondary
• Details from Experienced States
• Importance of 16 Issues
• Intentions as to Methods, Types of Skills, When
  Assessment Will Occur
• Numerator / Denominator in Transition Plan if
  Different than Guidance

7
Experience Among States
Using Technical Skill Assessments Secondary Postsecondary
Very Experienced no need to make alterations 5 1
Somewhat Experienced will need to make alterations 5 2
Experienced with some methods or program areas 8 14
Just Starting to Gain Experience or Familiarity 12 10
No Experience Whatsoever 5 7
Did Not Submit 17 17
8
Alternative to Guidance

• 23 of 35 States (65) Following Guidance on
  Technical Skill Attainment for Both Secondary and
  Postsecondary
• Secondary Exceptions 7 States
• 2 completion and 2 GPA (permanent or interim?)
• 3 specific changes year or numerator
• Postsecondary Exceptions 9 States
• 2 completion and 1 GPA (permanent or interim?)
• 2 waiting for data from colleges
• 3 specific changes to student definitions, year
  or numerator

9
Issues of Importance Top Tier

• Definition of Technical Skill
• Determining Phase-in / Interim Reporting
• Defining / Specifying Validity and Reliability
• Data Collection and Reporting
• End of Course / End of Program Assessments
• Development / Administrative Costs
• Value of Assessment to Student
• Student Definitions (Concentrator)
• Suitability of Assessment to Perkins
• Alignment to Standards
• All of these 60 Critical or Very Important

10
Issues of Importance Second Tier

• Distinguishing Among Foundational, Generic or
  Occupational-Specific Skills (may have confused
  w/ Definition of Technical Skill)
• Ability of and Devices for States to Collaborate
  and Pool Resources
• Dealing with Local v State Development or
  Selection of Assessments
• Developing a Seamless System for Secondary and
  Postsecondary
• Relationship Among Perkins Indicators
• Clarifying Clusters, Pathways and Programs of
  Study

11
Technical Skill AttainmentA Framework

• WHO needs to be measured?
• WHAT is to be measured?
• WHEN should measurement occur?
• HOW can measurement be accomplished?
• In general order of most uniformity to most
  flexibility

12
WHO Needs to be Measured?

• Student Definitions
• Concentrator
• Numerator / Denominator (also WHEN)
• Presented Day One (OVAE)
• Discussed Day One (Definitions) and Overarching
  Issues (Tech Skill Attainment Measure)

13
WHAT is to be Measured?

• Definition of Technical Skill
• Relationship of Technical Skill Attainment to
  Other Perkins Indicators Academic, Completion
• Distinguishing Among Foundational, Generic and
  Occupational Skills
• Clarifying Clusters, Pathways and Programs of
  Study
• Presented Day One (OVAE) and Two
• Discussed Breakouts One (S PS) and Two
  (Similar States)

14
WHEN Should Measurement Occur?

• Data Collection and Reporting
• Determining the Phase-In Period and Interim
  Reporting
• End of Course versus End of Program
• Developing a Seamless System for Secondary and
  Postsecondary (also HOW)
• Presented Day Two - Panels
• Discussed Breakouts One (S PS) and Two
  (Similar States)

15
HOW Can Measurement be Accomplished?

• Defining Validity / Reliability
• Alignment to Standards
• Suitability to Perkins
• Value to the Student
• Local v State Development / Selection
• Development / Administrative Costs
• Pooling / Sharing Resources
• Presented Day Two - Panels
• Discussed Breakouts One and Two

16
DQI Past and Present

• Quick Review of 05-06 Data Quality Institute
• Perkins IV Impact on
• Student Definition Concentrator
• Secondary Technical Skill Attainment Numerator
  and Denominator
• Postsecondary Technical Skill Attainment
  Numerator and Denominator
• Clarification on Overarching Issues
• Student Definitions (Recap of Day One)
• Phase-In Period and Interim Reporting How?
• Definition of Technical Skill
• Versus Academic Skill and Completion (Secondary)
• Versus Credential, Certificate or Degree
  (Postsecondary)