Developing Evaluation Instruments

1
Developing Evaluation Instruments
2
Evaluations

• Formative How are we doing?
• Summative How did we do?
• Confirmative How are we still doing?

3
Formative Evaluation

• Even excellent plans and concepts can be improved
• Gauge success by classroom program presentation
• Informs the instructor how well instructional
  package is serving the objectives as it progresses

4
Formative Evaluation

• Program of instruction is evaluated through
• Test results
• Learner reactions/comments
• General observation of learners during class
• Subject-matter expert reviews
• Colleague suggestions
• Quality control of the development process

5
Formative Evaluation

• Questions that may be asked
• Level of learning acceptable compared to
  objectives?
• Learners using knowledge to perform skills?
• Time element acceptable?
• Activities appropriate?
• Tests measuring objectives?

6
Summative Evaluation

• Measures success of major outcomes
• Usually follows all instruction, projects and
  testing in a program

7
Summative Evaluation

• Measures
• Learning efficiency (material mastered/time)
• Program cost (Development and delivery)
• Continuing expenses
• Student comments/reactions/evaluations
• Long-term benefit of program

8
Confirmative Evaluation

• Although instruction initially effective, some
  problems appear over time
• Tracks learner experiences over time to judge
  programs validity
• Relies upon numerous data-collection instruments
  (interviews, performance assessments, etc.)

9
Confirmative Evaluation

• Questions to investigate
• Do learners continue to perform correctly over
  time?
• Do materials still meet original objectives?
• How has technology and/or resources, trends, and
  attitudes changed since inception?
• How to best meet clients needs over time?

10
Confirmative Evaluation

• Changes needed in materials? What are the costs?
• If instruction not working as well as before
• Should instruction continue as is?
• Should it be revised?
• Should it be terminated?
• What might replace it?

11
Validity

• The test assesses what it is supposed to measure
  For example
• Performance tests assess processes and outcomes
  relating to skills or competencies
• Course attitude surveys need to measure reactions
  to the course

12
Reliability

• The test will produce consistent results whenever
  used
• More questions relating to specific objectives
  more reliable test
• Standardized delivery of test
• Scoring methods the less subjective, the better

13
Pretesting

• Judges the learners preparation to study the
  material (course or topic)
• Determines competencies already mastered

14
Testing for Prerequisites

• Standardized paper-and-pencil test
• Performance observations
• Questionnaire (Looks at learner background,
  training, and experiences)
• Review of past work experience
• Interview past supervisors/managers

15
Objective tests

• Multiple choice
• True or False?
• Matching items

16
Constructed-Response

• Make learner plan answers and express them in
  their own words
• Short-answer
• Essay questions
• Problem solving

17
Skills and Behavior

• Actions observed
• Process, product, or both?
• Constraints/limitations?
• Testing conditions simulated or realistic?

18
The Rubric

• General assessment of overall product
• Certain elements attain points
• More objective tool
• From Designing Effective Instruction 4th Ed.,
  Morrison, Ross, Kemp, 2004.