Designing the Test and Test Questions

1
Designing the Test and Test Questions

• Jason Peake

2
Principles of Testing

• Validity
• Does the test measure what it is supposed to
  measure?
• Does the test require student performance as
  described by the objective?

3
Principles of Testing

• Reliability
• Does the test generate the same results each time
  it is given?
• Is the test affected by misinterpretation
• Poorly written directions
• Vague questions
• Misleading words
• Not sure of response

4
Why do we evaluate students?

• Motivate students
• Measures achievement
• Identify areas for review
• Check effectiveness of materials, teaching
  method, and teacher
• Assign grades

5
Agenda

1. Aligning evaluation with level of learning.
2. Developing a test blueprint and preparing to
   write items
3. Common pitfalls when writing test questions
4. Writing questions at various levels of performance

6
Defining objectives using Blooms Taxonomy

• CATEGORY
• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

• SKILL
• Judge
• Combine
• Break apart
• Apply principles
• Understanding
• Recall

7
Sample Verbs

• Knowledge
• Cite, define, identify, label, list, match, name,
  recognize, reproduce, select, state
• Comprehension
• Classify, convert, describe, distinguish between,
  explain, extend, give examples, illustrate,
  interpret, paraphrase, summarize, translate
• Application
• Apply, arrange, compute, construct, demonstrate,
  discover, modify, operate, predict, prepare,
  produce, relate, show, solve, use

8
Sample Verbs continued

• Analysis
• Analyze, associate, determine, diagram,
  differentiate, discriminate, distinguish,
  estimate, infer, order, outline, point out,
  separate, subdivide
• Synthesis
• Combine, compile, compose, construct, create,
  design, develop, devise, formulate, integrate,
  modify, organize, plan, propose, rearrange,
  reorganize, revise, rewrite, tell, write
• Evaluation
• Appraise, assess, compare, conclude, contrast,
  criticize, discriminate, evaluate, judge,
  justify, support, weigh

9
Agenda

1. Aligning evaluation with level of learning.
2. Developing a test blueprint and preparing to
   write items
3. Common pitfalls when writing test questions
4. Writing questions at various levels of performance

10
Test blueprint (specifications)

• Matrix of number of test questions by topic and
  level of objective
• Helps you plan your test
• Do before writing items

11
Example blueprint

Topic A Topic B Topic C Topic D Total
Knowledge 1 2 1 1 5 (20)
Comprehension 2 1 1 2 6 (24)
Application 2 1 1 1 5 (20)
Analysis 1 1 2 2 6 (24)
Synthesis 1 1 2 (8)
Evaluation 1 1 (4)
Total 6 (24) 6 (24) 6 (24) 7 (28) 25 (100)
12
Agenda

1. Aligning evaluation with level of learning.
2. Developing a test blueprint and preparing to
   write items
3. Common pitfalls when writing test questions
4. Writing questions at various levels of performance

13
Preparing to write items

• Create test items while preparing class lessons
• Make note of questions asked frequently by
  students
• Make note of misconceptions made frequently by
  students during class or homework
• Invite students to submit items at the end of
  each class or at other times
• Pull questions from formative assessments.

14
Agenda

1. Aligning evaluation with level of learning.
2. Developing a test blueprint and preparing to
   write items
3. Common pitfalls when writing test questions
4. Writing questions at various levels of performance

15
Common pitfalls

• Providing unintended cues
• Using trick questions
• Using distractors that are obviously incorrect
• Using NOT or EXCEPT in the question
• Including all of the above, none of the above, or
  only A and B as option choices

16
Preparing for test-wise students

• Longest answer
• C as correct answer
• Avoid picking always or never
• Opposites
• Scientific sounding
• Avoid simple obvious
• Related word
• Guess

17
Agenda

1. Aligning evaluation with level of learning.
2. Developing a test blueprint and preparing to
   write items
3. Common pitfalls when writing test questions
4. Writing questions at various levels of performance

18
Levels of performance

• Lower levels (knowledge, comprehension, and
  application)
• Recognize the principle in altered format.
• Match component parts with vocabulary.
• Identify example seen before.
• Recognize an example never seen before.
• Solve a problem using familiar format, but
  different specifics.
• Higher levels (analysis, synthesis, evaluation)
• Pick out components of a situation.
• Problem solve by combining elements.
• Make judgments of value.

19
Categories of Test Questions

• Test can roughly be broken into
• Objective
• Multiple choice
• Matching
• Completion
• True false
• Subjective
• Short answer and/or Essay
• Oral test
• Case Study

20
Sample multiple choice items

• Lower levels
• Which of the following is an example of erosion?
  (choices include one or more used in class)
• Identify the definition of water erosion?
• Which of the following is an example of erosion?
  (choices never seen in class)
• Here is an erosion scenario. Which of the
  following principles apply?
• Higher levels
• Which of the following solutions would be the
  best to apply in the scenario described below to
  solve the erosion problem?
• Here is an erosion problem situation. Identify
  which principle from this class could be used to
  solve it.

21
Sample multiple choice distracters

• A term used in the unit but referring to a
  different concept
• Cause of erosion no till farming
• An everyday term that is close but misleading in
  meaning
• Cause of erosion hydrofoil
• A common misconception or confusion students have
• Cause of erosion lack of diverse soil particle
  size
• A plausible word or phrase that is meaningless
  but derives from something in the stem
• Cause of erosion systemic erosion syndrome

22
Matching

• Appropriate for only knowledge level testing
• Use a reasonable number of items 7 12
• All premises in one list, all responses in
  another list
• Premises can be long, but try to keep responses
  short
• Use more responses than premises

23
Clear Directions

• ____ 1. List premise
• ____ 2. List premise
• ____ 3. List Premise
• ____ 4. List Premise
• ____ 5. List Premise
• ____ 6. List Premise

1. Response
2. Response
3. Response
4. Response
5. Response
6. Response
7. Response

24
Completion Items

• Most useful for lower order levels of learning
• Use your own words
• Be sure only one answer can apply
• Keep all blanks the same length
• Avoid grammatical clues

25
Complete the meaning of each statement by writing
the correct word in the corresponding numbere
blank

• 1. A thermometer should be rinsed in
  _____________ water.
• 2. The NPK in fertilizers stands for
  _____________, _____________, and _____________

26
True / False Questions

• Can be used to check for knowledge and
  comprehension
• Keep questions simple
• Make entirely true or entirely false
• Avoid double negative items

27
Writing True / False Questions

• For each of the statements below circle T for
  true and circle F for false.
• T F Swine gestation period is 33 days.
• For each of the statements below circle T for
  true and circle F for false and explain in the
  blank why it is false.
• T F Swine gestation period is 33 days.
• _________________________________

28
Subjective Test Questions

• Short answer and/or Essay
• Oral test
• Case Study

29
Short Answer and/or Essay

• May require many different behaviors
• Comparison, for or against, cause and effect,
  explanation, summary, analysis, describe
  relationships, apply rules, or discussion

30
Oral Test

• All students must receive the same item stated in
  clear concise manner designed to elicit correct
  answers arranged in logical order.

31
Case Study

• Problem is practical and realistic
• Requires comprehension of prior learning and
  application of that learning

32
Subjective Test Questions

• Short answer and/or Essay
• Oral test
• Case Study
• Theses subjective test questions or subjective
  test require a criteria to be graded. We will
  work on grading criteria tomorrow.

33
Which Type of Question to Use?

• Select the appropriate technique to test the
  student competence at the level of cognition
  required by the objective.
• Avoid one type of test or test question
• We know students have different strengths
• Visual
• Audio
• kinesthetic

34
Materials in this presentation developed in part
by

• Brian Parr
• University of Georgia
• Dr. Dawn Zimmaro
• University of Texas