Item Analysis

1
Item Analysis

• Reference Anastasi, A. (1988). Psychological
  testing, 6th edition. New York Macmillan
  Publishing Co.

2
Item Difficulty

• Item difficulty is used to choose items of a
  suitable difficulty level
• Percentage Passing The difficulty of an item is
  usually expressed in the percentage of persons
  who answer it correctly
• Tests are usually arranged with items in order of
  difficulty, beginning with easier items

3
What are the ranges for item difficulty?

• Item difficulty (p) ranges from 0 to 1.0
• A zero means that no one got the item right,
  while a 1.0 means that everyone got the item
  right
• The closer an item gets to 0 or to 1.0, the less
  information it contributes about test takers

4
What levels of item difficulty are desirable in a
test?

• It is best to chose items close to p.5, however
  if the items within a test are highly
  intercorrelated (the test is more homogeneous),
  then there should be a wider range of item
  difficulties (but they should average 0.5)

5
Relating Item Difficulty to Testing Purpose

• The level of item difficulty required depends, in
  part, on the purpose of the test
• For screening tests, the item difficulty should
  be close to the desired selection ratio
• For example, if you want to select the upper 30
  of the cases, then p.30
• If the purpose is to test for mastery, then p is
  set higherprobably around .80 or .90

6
Item Discrimination (D)

• Item discrimination refers to whether an item can
  distinguish between people who scored high or
  scored low on a test
• This is calculated by a correlation between each
  item score and the total test score
• Item score calculation
• 1 (if the item was answered correctly)
• 0 (if the item was answered wrong)
• A point biserial correlation is used (special
  case of Pearson)

7
Shortcut for Calculating D

• 1. Divide group of examinees in half
• 2. Count in the high group who got item right
  RH
• 3. Count in low group who got item right (RL)
• Then, D RH - RL /(0.5N)
• As an alternative, if N is very large, then use
  the top 27 and the bottom 27 and leave out the
  middle portion

8
Example

• If N 100 and everyone in the high group gets
  the item right
• D (50-0)/50 1
• This is the highest that D can be
• If D 1, then the item perfectly discriminates
  between high and low scoring groups

9

• If D -1, then the item doesnt discriminate at
  all. It tells you that something is wrong with
  the item (people who scored high on the test,
  missed that item).
• Whenever you get a negative value for D, the item
  needs to be reviewed

10
What Values of D are Desirable?

• To maximize internal consistency we want items
  with a value of D close to 1.0. We want to
  eliminate items with a value close to 0 as well
  as negative items
• Relationship between p and D
• If p .5, then D is close to 1.0

11
Distractor Analysis

• Distractor analysis looks at a proportion of
  examinees who choose a certain distractor in
  multiple choice tests
• This analysis tells you what types of mistakes
  are being made
• In looking at a discrimination index for
  distractors, you want to get a negative value. It
  is desirable to have a negative correlation
  between people who scored high and also chose
  that distractor

12

• Conversely, there should be a positive
  correlation for the correct choice
• This means that people who scored high on the
  test chose the correct response for a particular
  item
• (Example of distractor analysis on handout)

13
Item Response Theory (IRT)

• Also called Latent Trait Theory, Item
  Characteristic Curve (ICC) Theory

14

• IRT gives information about how an item functions
  within a test
• It takes ability levels into account
• Published information about item analysis can
  help test users in their evaluation of these
  tests
• Using item analysis, test developers can improve
  the reliability of their instruments

15

• Item-test regression combines item difficulty
  and item discrimination in the same graph
• This allows us to visualize how effectively an
  item functions

16
Item test regression for items 7 (orange) and 13
(purple)
17
Basic Features of IRT

• Item performance is related to the estimated
  amount of the respondents latent trait (this
  is a statistical conceptnot a psychological one)
• Item characteristic curves (ICCs) are plotted
  from mathematically derived functions. Different
  IRT models use different assumptions

18
Examples of ICCs

• ICCs for 3 items
• The item discrimination parameter indicates the
  slope of the curve
• A more gradual slope (item 3-green) has a lower
  discriminative value

19
How does the ICC relate to item bias?

• A set of items is judged unbiased if the ICCs for
  every item are the same for both groups.
• This is because irrelevant sources of variance
  (as opposed to item bias) will affect both groups
  the same way
• Unequal ICCs give evidence of item bias

20
Advantages and Disadvantages

• ICC scaling allows for test-free measurement you
  can compare people on a trait even if they
  answered different test items
• However, a large sample (approx. 1000) is needed
  to estimate the item parameters