CAREERTECHNICAL EDUCATION AND ASSESSMENT

1
CAREER-TECHNICAL EDUCATION AND ASSESSMENT

• PAST, PRESENT, POSSIBLE

2

• James T. Austin
• Assessment and Evaluation Services
• Center on Education Training for Employment
• College of Education
• The Ohio State University Big 10 Champions

3
AGENDA

• Assessment
• Defined
• Two Worlds Large-Scale Classroom
• Perspectives
• Past Getting To The Way We Were
• Present High Stakes Testing
• Possible Expansions In Multiple Directions
• Example Applications Using Graphic Designer

4
TWO CULTURES

• Large Scale Assessment (district level up)
• What states/districts/nations do
• Prototype of assessment to public
• Few players (NCS, Harcourt, CTB/McGraw-Hill)
• Norm- and/or criterion-referenced assessments
• Quality control known and used (reliability,
  validity)
• Classroom Assessment
• What teachers do
• Phyes handbook covers this domain thoroughly
• Standards of assessment competence are available

5
ASSESSMENT TRIANGLE

• Cognition a feature of thinking or performing
• Knowledge representation, skill development
• EX mental models of software project, artistic
  competence
• How do mental models change as novices become
  proficient?
• Observation a situation and stimulus
• Task situations that require observable responses
• EX entry-level tasks as a part of graphic
  design projects
• Design creation of brochures for an ad campaign
• Interpretation inference linking
  cognition-observation
• Methods for scoring thinking/performing from
  observations
• EX statistical model for performance assessments

6
PURPOSES SERVED

• Assessment conducted to
• Assist learning
• Measure individual achievement
• Evaluate programs

7
SAMPLES OF ASSESSMENT

• Think of CTE students
• 1. in class in the graphic design lab
• What is the error in this brochure layout? Can
  you fix it?
• 2. in teams at SkillsUSA/VICA competition
• Design a logo for Organization X under
  constraints.
• 3. answering high stakes writing prompt
• Persuade school board to adopt a stand on X.
• 4. taking a graphics certification exam on-line
• Modify given artwork according to change ticket.

8
ASSESSMENT PASTHOW DID WE GET HERE ?

• Tests used throughout history
• Biblical Chinese examples
• Binet, Army alpha/beta tests marked start
• Expansion of testing as technology 1920-2000
• Military testing for recruit selection
  placement
• Educational testing to assess achievement
  (Resnick)
• Business testing for personnel selection
• Certification testing for a range of professions
• Testing is a social institution (Hanson, 1993)

9
CTE TESTING PAST

• Trade tests developed during WW I
• differentiated novices, apprentices, experts
• Permitted evaluation of claims of recruits
• Prosser and Quigley well aware of testing
• Vocational Education in Democracy (1949)
• Intelligence, trade, and entry tests defined
• Testing varies for entry vs experienced

10
O.T.A. 1994 REPORT

• Comprehensive report in parts
• Classified state policies for skills assessment
• 18 states mandated local written testing
• 15 states mandated testing without guidance
• 10 states encouraged testing guidance
• 8 states had no specific policy
• Assessment targets
• Academic occupational (vocational, generic
  workplace, broad technical skills)

11
O.T.A. CONCLUSIONS

• (1) Assessment in secondary CTE differed best
  practices like alternative forms of assessment
  just being explored for rest of education but
  quality varies greatly.
• (2) In CTE, testing integral to learning
• (3) Critical issues in performance assessment
• Comparability of judgments
• Judgment-standard correspondence
• (4) Critical issues in written testing
• Relevance of items to capabilities for
  performance
• Effects of testing on teaching learning

12
ASSESSMENT PRESENT

• Testing now major part of accountability
• Systems concepts key to appropriate high stakes
  testing
• Linn (2001) models design and evaluation of such
  systems reliable, valid, fair
• Standards for Educational Accountability criteria
• National Board for Educational Testing Public
  Policy as an independent auditor

13
ASSESSMENT PRESENT

• Testing increasing for some time into future
• No Child Left Behind Act
• Annual testing Grades 3-8, interventions/consequen
  ces
• HST increasing for students and teachers
• Making Standards Matter (AFT, 2001) documents
  prevalence
• Miller et al. (2001) review teacher testing for
  NRC
• Threat or opportunity? Tradeoffs?
• Opposition, but Cizek Phelps believe HST is an
  opportunity

14
PRESENT

• Alternative models seek full status in both
  worlds
• Stecher et al. (1996) reviewed for CTE field
• Custer (2000) also surveyed for CTE field
• Vermont, Maryland, Kentucky
• Koretz et al. (1991) reviewed Vermont large-scale
  experience with portfolios
• Herman (1997) reviewed large scale alternative
  assessment

15
PRESENT

• Technology always present, increasing
• Testing in general
• Bennett (1999)
• Distance assessment
• Kingsbury Houser (1999) discuss one system
• Major testing firms rolling out (Sylvan
  Prometric)
• Much more
• Scoring reporting facilities
• Roll-up for Perkins III Consolidated Annual Report

16
LARGE-SCALE CTE PRACTICES

• HSTW ASSESSMENT
• Used to monitor progress of this reform since
  1987
• Operated by ETS for SREB (given in even years)
• Modeled on standards in Math, Language, Science
• NOCTI
• Long-time player (since 1973)
• Combines written performance formats
• Student and teacher competency assessments
• Now evaluating on-line assessment systems

17
TYPICAL CTE PRACTICES

• DISTANCE DELIVERY
• Mostly first generation, meaning NOT adaptive
  testing
• Several systems use QuestionMark software
• OKLAHOMA
• www.okcareertech.org/testing/online.htm
• OHIO
• Tech-Prep modular assessment under development
• ITWorks standards represent collaboration between
  government business (www.itworks-ohio.org)
• Modular organization, 30 items per module
• Core modules cut across 4 pathways or strands
• IM, ISS, NS, PSD are the occupational areas
• Other module content is specific or cuts across a
  subset (1-3)

18
THEMES PAST PRESENT

• Dissent since 1920s (Madaus, 1995, 1997)
• Tyranny of testing in 1962
• Protests increasing (Sacks, Kohn, Popham)
• Reforms
• Competency testing 1970s (Haney, 1978)
• Standards to fore 1990s (Resnick Wirt, 1996)
• High Stakes Testing (Heubert Hauser, 1999)

19
THEMES

• Standards for educational psychological testing
  (1999)
• Just revised by 3 professional groups
• Ch 13 addresses educational applications
• 19 standards (validation for each purpose)
• Validation as consequences intended/unintended
• High stakes tests retention
• Industry certification tests student engagement

20
ASSESSMENT POSSIBLE

• What might 21st century bring to CTE?
• Possible futures applied to graphic design
• rethinking redesigning CTE assessment
• alternative assessments
• computerized distance assessment

21
Scenario 1 RETHINK

• Represents most change in way things are done
• Has the most scientifically-based support, also
  costs the most
• National Research Council review panel
• Pellegrino et al. (2001) Knowing what students
  know
• Merges cognition, measurement, technology
• Cognition, observation, interpretation form
  assessment triangle
• Called for 3 types of alignment
• vertical (levels of educational system)
• horizontal (across curriculum, instruction,
  assessment)
• temporal (across students educational career)
• Tennessee VAAS is a model, with some flaws

22
APPLICATION

• Need assessment triangle C-O-I
• COGNITION
• Schemas, mental models, expert-novice
• OBSERVATION
• Create situations responses to tap
• INTERPRETATION
• Select models to link C O

23
APPLICATION TO CTE

• Details require an occupation
• I chose graphic design
• Fortuitously, Techniques March issue arrived
• Consciously looking for a sample occupation
• Emerging, entrepreneurial
• Traditional (advertising) and IT (website design)
• Blend of artistic technology inputs
• Next slides provide background

24
GRAPHIC DESIGNERS

• Design/create graphics to meet specific
  commercial or promotional needs, such as displays
  or logos use a variety of mediums to achieve
  artistic or decorative effect (ONet-SOC Code
  27.1024.00)
• Work domains prepress, press, binding

25
GRAPHIC DESIGN

• National Council for Skill Standards
  (www.ncss.org)
• Used common framework of National Skill
  Standards Board
• Standards certifications available
• Electronic Imaging (pre-press)
• Sheetfed and Web Offset Press (press)
• Flexographic Press (press)
• Finishing and Distribution (binding)

26
SOME PROGRAMS

• Secondary Metro Tech, OK Thomas Edison, MD
• Post-secondary Tunxis CC, CT
• Extended Cuyahoga Valley Career Center, OH
• Digital Design Program (Exemplary program, 2000)
• Interactive Media pathway of Tech-Prep IT program
• 2 years (48 students), screened by
  guidance/recruiting team
• Art plus computers powerful combination
• Look at the waiting lists!!
• 100 retention rate, 76 placement rate

27
POSSIBLE

• Microworlds" built for the graphic designer
• (i.e., a simulated work environment with
  individual team elements that an entire class
  would enter for a semester or so)
• Portfolios of student work
• Across the (usually) 2 years of a program
• Performance assessments
• start with smaller tasks of entry-level
  professional synthesize to integrated project
  level

28
POSSIBLE

• Some drawbacks
• Many models are fairly domain-specific
• Will require extensive occupational analyses
• Transitions in Work and Learning (NRC, 1997)
• Construct/validate mental models, contextual
  situations
• Development of the C-O-I assessment triangle
  elements
• Creation of standards curriculum

29
Scenario 2 ALTERNATIVE ASSESSMENT

• Wiggins has been a consistent advocate
• Authentic (1989 book) educative (1998 book)
• Some alternative assessments in CTE
• (1) Career-Technical Assessment Program
• (2) Kentucky Instructional Results Information
  System
• (3) Laborers-AGC Environmental Training
  Certification
• (4) National Board for Professional Teaching
  Standards
• (5) Oklahoma Competency-Based Testing Program
• (6) VICA National Competition
• (7) NOCTI TOCT SOCAT

30
WIGGINS AUTHENTICITY

• Authenticity derives from match of assessments
  to
• (1) "worthy problems or questions of importance"
• (2) faithful representation of contexts facing
  workers in a field
• (3) non-routine, multi-stage tasks
• (4) tasks that require generating a quality
  product, performance
• (5) transparent or demystified criteria
• (6) interactions between assessor and assessee
• (7) effect of process product/performance
  determines quality
• (8) trained assessor judgment
• (9) the search for patterns of response in
  diverse settings
• Tradeoffs associated with achieving authenticity

31
AUTHENTIC

• Applied Psychological Measurement
• Performance assessment special issue (Hambleton,
  2000)
• Miller Linn on validation
• Hambleton, Jaeger, Plake on standards
• Clauser on developing scoring

32
POSSIBLE

• Graphic Design uses portfolios
• Would portfolios change?
• Additional assessments?
• Performance assessments
• Projects
• Presentations (to advisory committee??)

33
Scenario 3COMPUTER-DISTANCE ASSESSMENT

• Computers used to deliver assessments since
  1960s, especially influential in adaptive testing
  and format expansions (multimedia)
• Emergence of networks brings distance delivery to
  the forefront
• RAND Issue Paper (Klein Hamilton, 1999)
• NGA position paper on distance assessment (2002)

34
COMPUTERIZED

• Curriculum instruction
• Evolution from correspondence and TV/video
  courses to distance education
• Assessment
• Evolution from individual group paper-pencil to
  authentic, computer networked
• Computer performance assessment of
  problem-solving
• ONeil (1999) special issue of Computers in Human
  Behavior
• Combines authentic computer/distance
  assessments
• Consider graphic design scenario

35
POSSIBLE

• Graphics Design profession
• SkillsUSA evaluates 5 areas
• layout and design
• film image assembly
• plate making
• offset duplication
• written test of common graphic comm issues
• Contestants compose final art using a desktop
  computer and software system

36
POSSIBLE

• Graphic Design application
• Extracted from British distance course, web-site
  design (http//www.it.bton.ac.uk/staff/lp22/CP303)
  
• Deliverable 1 an initial requirements
  description
• Deliverable 2 brief analysis of comparable sites
  
• Deliverable 3 user class analysis, in the form
  of personas
• Deliverable 4 task scenarios
• Deliverable 5 user object model
• Deliverable 6 content structure diagram
• Deliverable 7 site structure diagram
• Deliverable 8 initial evaluation results
• Deliverable 9 storyboards
• Deliverable 10 lo-fi prototype
• Deliverable 11 evaluation report
• Deliverable 12 maintenance document
• Deliverable 13 design document

37
HOW IS ASSESSMENT DONE?

• WestEd distinction from CTAP
• Cumulative developed over time
• On-demand real-time, in the spotlight
• Portfolios are cumulative
• Useful internally (student evaluation)
• Useful externally (job searches)
• Other formats can be on-demand
• Sudden presentation to client

38
HOW COULD WE ASSESS ?

• Three Possible Futures
• Rethinking redesigning
• Alternative assessments
• Computerized distance assessment
• Apply to Both Worlds of Assessment

39
CONCLUSIONS

• Assessment for CTE viewed from several
  perspectives
• Two worlds of classroom and large-scale
  assessment
• History past, present, potential
• Exemplary programs of assessment
• What are some important goals to be debated?
• Can/should the two worlds be blended?
• What constitutes scientifically-based research
  support?
• How can tradeoffs be considered?
• Balance claimed gains from new assessments
  against costs
• What are actions needed to reach consensual
  goals?
• Assessment literacy for CTE teachers and
  administrators

40
ASSESSMENT LITERACY

• Concept articulated by Richard Stiggins
• Lacking for many educators (1991), but an immense
  potential upside (2000) for both worlds
• Large scale buy-in and preparation for students
• Classroom assessment improvements
• Teachers what capabilities are required?
• Leading edge
• Administrators what capabilities are required
• Support and guidance