Developing Assessment Instruments

1
Developing Assessment Instruments

• Dick Carey
• Chap. 7

2
Criterion-Referenced Tests

• Designed to measure explicit behavioral
  objectives
• Used to evaluate
• learner performance
• effectiveness of the instruction

3
Criterion-Referenced

• Also called objective-referenced
• Refers directly to explicit criterion or
  specified performance
• Criterion-Referenced Test must
• match test item and performance objective
• stipulate degree of mastery of the skill

4
Types of Criterion Tests

• Pretest
• 1. Consists of items that
• measure entry behavior skills
• test skills to be taught
• draw from skills below the entry behavior line
• 2. Helps determine appropriateness of required
  entry skills.
• 3. Used during formative evaluation process. May
  be discarded in final version of instruction.

5
Types of Criterion Tests

• Posttest
• 1. Assesses all the objectives, focusing on
  terminal objectives
• 2. Helps identify ineffective instructional
  segments
• 3. Used during the design process and may be
  eventually modified to measure only terminal
  objectives

6
Designing Tests for Learning Domains

• Intellectual Verbal Information
• paper pencil
• Attitudinal
• state a preference or choose an option
• Psychomotor
• performance quantified on checklist
• subordinate skills tested in paper-and-pencil
  format

7
Determining Mastery Levels

• Approach 1
• mastery defined as level of performance normally
  expected from the best learners
• arbitrary (norm-referenced)
• Approach 2
• defined in statistical terms, beyond mere chance
• mastery varies with critical nature of task
• example nuclear work Vs. paint a house

8
Writing Test Items

• What should test items do?
• Match the behavior of the objective
• Use the correct verb to specify the behavior
• Match the conditions of the objective

9
Writing Test Items

• How many test items do you need?
• Determined by learning domains
• Intellectual requires three or more
• Wide range use random sample

10
Writing Items (continued)

• What types (true / false, multiple choice, etc..)
  to use?
• clues provided by the behavior listed in the
  objective
• review Types of Test Items this chap. p 148

11
Writing Items (continued)

• Item types tempered by
• amount of testing time
• ease of scoring
• amount of time to grade
• probability of guessing
• ease of cheating, etc.
• availability of simulations

12
Writing Items (continued)

• What types are inappropriate?
• true / false for definition
• discrimination, not definition
• Acceptable alternatives from best possible
• for simulations
• list steps

13
Constructing Test Items

• Consider
• vocabulary
• setting of test item (familiar Vs. unfamiliar)
• clarity
• all necessary information
• trick questions
• double negatives, misleading information, etc.

14
Other Factors

• Sequencing Items
• Consider clustering by objective
• Test Directions
• Clear and concise
• General
• Section specific
• Evaluating Tests / Test Items

15
Measuring Performance, Products, Attitudes

• Write directions to guide learner activities
• and

16
Evaluating Performance, Products, Attitudes

• Construct an instrument to evaluate these
  activities
• a product, performance, or attitude
• Sometimes includes both process and a product
• For example -- TRDEV 518

17
Test Directions for Performance, Products,
Attitudes

• Determine the
• Amount of guidance?
• Special conditions
• time limits, special steps, etc.
• Nature of the task (i.e., complexity)
• Sophistication level of the audience

18
Assessment Instruments for Performance, Products,
Attitudes

• Identify what elements are to be evaluated
• cleanliness, finish, tolerance of item, etc.
• Paraphrase each element
• Sequence items on the instrument
• Select the type of judgment for rater
• Determine instrument scoring

19
Formats for Assessments of Performance,
Products, Attitudes

• Checklist
• Rating Scale Frequency Counts
• Etc.

20
Performance, Products, Attitudes -- Scoring

• Guidelines?

21
Evaluating Congruency

• Skills, Objectives, Assessments should refer to
  the same behaviors
• To check for congruency
• Construct an Congruency Evaluation Chart
• include Subskills, Behavioral Objectives,
  Test Items