DIFFERENTIAL EXPECTATIONS OF STUDENT PERFORMANCE

1
DIFFERENTIAL EXPECTATIONS OFSTUDENT PERFORMANCE

• NACTEI Conference 2008
• Boise, Idaho
• Dr. Paul Munyofu

2
Past and Present Requirement

• All students who reach a state defined threshold
  level of vocational education will master the
  knowledge and skills that meet state established
  industry standards
• Perkins III and IV (2S1)

3
Measurement

• Numerator Number of career and technical
  education concentrators who perform at or above
  the Competent level on PDE approved occupational
  tests in the reporting year.
• Denominator Number of career and technical
  education concentrators who complete the PDE
  approved occupational tests in the reporting year.

4
Related Requirement

• All students who reach a state-defined level of
  education will attain a proficiency credential in
  conjunction with a secondary school diploma or
  its state equivalent
• Perkins III (2S2), Perkins IV (3S1)

5
Measurement

• Numerator Number of CTE concentrators who earned
  a regular secondary school diploma, or earned a
  proficiency credential, certificate, or degree,
  in conjunction with a secondary school diploma
  during the reporting year.
• Denominator Number of CTE concentrators who left
  secondary education during the reporting year

6
PA Credential Measurement

• Numerator Number of career and technical
  education students who achieve competency levels
  at or above the Advanced (PSC) on approved
  Job-Ready End-of-Program tests.
• Denominator Number of career and technical
  education students who complete the approved
  Job-Ready End-of-Program tests.

7
Job-Ready Tests

• Occupational Assessments
• Industry Aligned
• End of Program
• Specific to CIP
• Both Written and Performance
• NOT to be confused with
• Workplace Readiness (NOCTI)
• Work Keys (ACT)

8
Assessments Utilized

• NOCTI, ACF, AWS, ICE, AYES
• NIMS, ASE, NATEF, CompTIA
• State Licensures (COS, LPN, NA)

9
State Standard Setting

• Levels of Performance
• Advanced
• Competent
• Basic
• Below Basic

10
Advanced, Competent

• Advanced Level This level reflects mastery of
  competence and understanding of academic/career
  and technical skills and knowledge required for
  advanced placement in employment and/or
  postsecondary education.
• Competent Level This level reflects a solid
  acquisition of academic/career and technical
  skills and knowledge required to enter employment
  and/or postsecondary education

11
Basic

• Basic Level This level reflects an adequate
  attainment of academic/career and technical
  skills and knowledge required to enter employment
  or postsecondary education. Students with this
  score would function at an entry level, but
  would require some assistance on the job.

12
Below Basic

• Below Basic Level This level reflects a
  partial acquisition of skills and knowledge
  needed to perform a given assignment, task or
  operation on the job. Additional instruction
  and/or assistance are necessary in order for the
  student to successfully complete specific
  assignments. Students with this score did not
  acquire the minimum skills required for the
  occupation.

13
The Nedelsky Model

• Present a Multiple-Choice Question to a Subject
  Matter Expert.
• In the mind of a new-hire who is Competent for
  employment
• HOW MANY OF THE k ITEM CHOICES CAN THIS
  INDIVIDUAL ELIMINATE BEFORE RESORTING TO GUESSING
  FROM THE (k j) REMAINING CHOICES?

14
Probabilities from 4 Choices

Number Eliminated (j) Probability Expected
0 0.25
1 0.33
2 0.50
3 1.00
15
Probabilities from 5 Choices
Number Eliminated (j) Probability Expected
0 0.20
1 0.25
2 0.33
3 0.50
4 1.00
16
Training and Practice

• Pennsylvania Drivers Examination
• Practice Test

17
Practice

• 1. What kind of crash is most likely to cause
  injury?
• a. red light running
• b. left-turn crashes
• c. rear-ending a stopped or stopping vehicle
• d. running off the road

18
Practice

• 2. When driving on slick roads, you should
• a. take turns more slowly
• b. accelerate quickly
• c. brake hard
• d. change lanes quickly

19
Practice

• 3. Hydroplaning is usually caused by
• a. excessive speed
• b. sudden turns
• c. sudden stops
• d. excessive stops

20
Practice

• 4. What is the greatest danger factor for teen
  drivers?
• a. lack of skill
• b. risk taking
• c. ignorance
• d. alcohol and drugs

21
Practice

• 5. What is the leading public health problem for
  young people aged 13 to 19?
• a. AIDS
• b. motor vehicle crashes
• c. gun violence
• d. cancer

22
Practice

• 6. Allowing a space cushion is important because
  it
• a. keeps other drivers alert
• b. prevents distractions from other vehicles
• c. keeps traffic flowing at a safe pace
• d. allows you time to react to situation

23
Practice

• Correct Answers
• 1. c
• 2. a
• 3. a
• 4. d
• 5. b
• 6. d

24
Results

• Judgments from two tests
• Precision Machining
• Early Childhood Care

25

26
Item J 2(Ind.) J 3(Ind.) J 7(Ind.)
1 1.00 1.00 1.00
2 1.00 1.00 0.50
3 1.00 1.00 0.50
4 1.00 1.00 1.00
5 1.00 0.50 0.33
6 1.00 0.25 0.33

27
J 1(Instr.) J 4(Instr.) J 15 (Instr.) MeanInd. MeanInstr.
0.33 1.00 1.00 1.0000 0.6550
0.50 0.50 0.33 . 0.8750 0.6038
1.00 0.33 0.50 . 0.8750 0.5613
1.00 1.00 0.50 . 1.0000 0.7288
1.00 0.50 1.00 . 0.7075 0.6038
0.33 0.33 0.33 . 0.5200 0.3938

28
27 0.50 0.50 0.25
28 1.00 0.25 0.25
29 1.00 0.50 0.50
30 0.25 0.25 1.00
31 0.25 0.25 0.25
. . . .
171 1.00 0.50 0.50
Total 153.33 111.97 110.49
Mean 0.90 0.65 0.65
29

0.50 1.00 0.50 . 0.5625 0.5413
0.25 1.00 1.00 . 0.6250 0.5313
1.00 1.00 0.50 . 0.7500 0.6663
0.25 0.33 0.50 . 0.6250 0.3850
0.25 0.25 0.25 . 0.2700 0.2813
. . . . . .
0.33 0.33 0.33 . 0.7500 0.4775
91.04 107.35 80.51 . 130.10 97.19
0.53 0.63 0.47 . 76.08 56.84
30
(No Transcript)
31
Test Item J 1 (Ind.) J 3 (Ind.) J 9 (Ind.)
1 1.00 0.33 1.00
2 0.33 0.33 0.25
3 0.50 0.33 1.00
4 1.00 0.33 1.00
5 1.00 1.00 0.25
6 0.33 0.33 0.33

32
J 2 (Instr.) J 12 (Instr.) J 14 (Instr.) Mean (Ind.) Mean Instr.
. 0.50 1.00 0.50 . 0.8225 0.6667
. 0.33 0.50 0.25 . 0.3413 0.3733
. 1.00 1.00 0.33 . 0.6450 0.8050
. 0.50 1.00 0.25 . 0.9163 0.7917
. 1.00 0.50 0.33 . 0.7813 0.6100
. 0.33 0.33 0.33 . 0.3300 0.3583

33
27 0.33 0.33 0.33
28 0.33 0.33 0.50
29 1.00 1.00 0.25
30 0.50 0.50 1.00
31 1.00 0.33 1.00
. . . .
193 0.50 0.33 0.50
Total 109.08 90.53 149.28
Mean 0.57 0.47 0.77
34

1.00 0.50 0.33 . 0.4150 0.5267
0.25 0.50 0.33 . 0.3950 0.3733
1.00 0.25 0.25 . 0.8125 0.5417
0.50 0.33 0.25 . 0.5625 0.3600
1.00 0.50 0.33 . 0.7913 0.7217
. . . . . .
0.33 0.50 0.33 . 0.3738 0.3583
115.20 140.75 78.66 . 119.30 110.68
0.60 0.73 0.41 . 61.82 57.35
35
Results

• Judgments from 17 tests
• Industry Mean
• Instructor Mean
• Two-tail Significance
• Correlation Coefficient

36
Table 3. Comparative Data on Mean Cut Scores for
Selected NOCTI Written Tests
Test Title Industry Mean Instructor Mean Significance Correlation
Advertising and Design 60.99 57.07 0.56 .66
Automotive Technician 52.52 49.52 .06 1.00
Building Construction Occupations 73.60 54.69 6.01E-27 .42
Business Information Processing 69.17 61.32 2.60E-8 .56
Carpentry 51.90 51.81 .94 .47
Commercial Foods 86.47 63.17 6.01E-27 .42
Computer Technology 51.44 51.91 .96 1.00
Cosmetology 47.23 56.17 1.01E-8 .68
Early Childhood Care and Education 61.82 57.35 3.46E-6 .70
Electrical Occupations 61.12 50.05 1.83E-23 .69
Food Production 76.62 71.37 1.98E-5 .16
Graphic Communications 62.68 62.14 .80 .31
Nurse Assisting 67.63 65.85 .24 .68
Precision Machining 76.08 56.84 1.26E-33 .54
Production Agriculture 64.75 48.99 4.7E-26 .57
Retail Trades 73.63 72.24 .28 .57
Welding 54.56 64.39 4.82E-12 .34

37
Results

• Instructors, generally, tended to set a lower
  predicted cut score than their industry
  counterparts.
•  
• Many instructors said they were influenced by
  their own students.
•  
• Claim CTE was a dumping ground for students who
  were characteristically educationally-challenged
•  

38
Results

• Response
• Under Perkins, all students who finish an
  approved career and technical education program
  are expected to be ready for college and work.

39
Results

• Claim
• NOCTI tests were outdated, showing little
  relevance to the present status of industry.
•  

40
Results

• Response
• 2 3 year Revision Cycle on all tests
• National subject matter experts involved in test
  development
• All aligned to national industry standards.

41
Results

•  
• Claim
• NOCTI assessments are relatively unknown.

42
Results

• Response Not any more!
• A vast array of services that the organization
  provided to their clients. The data contains
  individual, class, school, state and national
  information that can be used to evaluate and
  improve career and technical education programs.

43
Results

• Claim
• Some schools have a large percentage of special
  needs students enrolled in CTE.

44
Results

• Response
• Perkins IV has stressed the importance of wedding
  career and technical education with rigorous
  academics. It is the expectation that all
  students who complete career and technical
  education programs be prepared for postsecondary
  education and work.

45
Results

• NOCTI also provides assessments for experienced
  workers and for potential teachers.
• Missing is a postsecondary component.
•  
• Recommended that NOCTI develop tests that fill
  the gap and attain the Perkins gold standard.

46
Implications

• Standard-setting is applicable anywhere there is
  an valid assessment.
• Criteria for establishing levels of performance
  have to be clearly related to the graduates
  ability to work in their chosen industry.

47
Recommendations

• Create or increase local Industry Partnerships
  for curriculum development, program evaluation,
  other assistive services.
•  
• Initiate an articulation of national performance
  levels, with corresponding cut scores that have
  the same meaning.

48
Contacts

• Dr. Paul Munyofu
• 717-783-6623
• pmunyofu_at_state.pa.us
• www.pde.state.pa.us/bcte/perkins