Psychometric Theory

1
Psychometric Theory

• Seminar in Vocational Assessment
• (BCE 545)
• March 4, 2002

2
What is a test?

• A way of measuring something.
• A means for gathering information.
• A means for evaluating performance
• A means for making decisions
• Psychometric (mind-measure) Testsmeasure traits
  of individuals that relating to emotion and
  cognition.

3
What makes a good test?

• A good test should be
• Valid
• Reliable

4
Test Validity
5
What is Validity?

• Validity is the extent to which a test actually
  measures the attribute for which it is designed.

6
Types of Validity

• Construct Validity
• Content Validity
• Predictive Validity
• Criterion-Related Validity
• Incremental or Criterion-Related Validity
• Face Validity

7
Construct Validity

• Is the operant definition of the trait being
  measured legitimate?

8
Content Validity

• Is the test broad enough to measure all of the
  components of the construct (trait) being
  measured?

9
Predictive Validity

• Is the test capable of predicting performance in
  observable events?

10
Incremental or Criterion-Related Validity

• Similar to predictive validity, criterion-related
  validity refers to the ability of a test to
  predict success/failure through cutoff scores
  or categories.

11
Face Validity

• Does the test look like it is valid?
• (especially important with work samples)

12
Measuring Validity

• Validity is addressed in the tests
  administration or users manual.
• Construct/content validity is addressed through
  documenting the tests constructioncitations,
  definitions, descriptions of theory, etc.
• Incremental Validity can be measured
  statisticallywith a validity coefficient
  (.01-.99)
• Face ValidityIn the eye of the beholder.

13
Reliability
14
Reliability

• Is an individuals performance on a test
  consistent over time?
• An individuals performance should remain
  constant on a valid test provided that testing
  conditions are consistent and no new learning has
  taken place.
• Tests that are valid are also, by necessity,
  reliablebut tests that are reliable are not
  necessarily valid.

15
Types of Reliability

• Split-Half Reliability (measure of internal
  consistency of a test)
• Test-Retest reliability
• Test-Retest with alternate forms of the test.

16
Split-Half

• An individuals scores on even-numbered items is
  correlated with scores on odd-numbered items.
• Statistic Used Cronbachs Alpha (correlation
  coefficient).

17
Test-Retest Reliability

• Test is administered to a group of subjects and
  readministered within 1-2 weeks. Scores of the
  two tests are correlated.
• Sometimes uses parallel forms of test.
• Test Statistic Kuder-Richardson correlation
  coefficient (usually KR-20 or KR-21)

18
Assessing Reliability

• Correlation coefficient (-1.0 - 1.0)
• For reliability, reliability coefficient is a
  score between 0 and 1.
• How good is the reliability coefficient?
• 0.80-1.00 Very High Correlation
• 0.60-0.79 Substantial Correlation
• 0.40-0.59 Moderate Correlation
• 0.20-0.19 Little Correlation
• 0.00-0.19 No Correlation

19
Considering the value or appropriateness of a test

• Investigate the validityespecially the
  constructs on which the test is based. Is it
  well researched and constructed?
• Look at the reliabilityis the test consistent?
• View the test instrument itself, or better, take
  the testwhat is the face validity?
• Is the test appropriate for the
  individual/population you are assessing?

20
Norms and Standardization
21
What is standardization?

• Everyone who takes a test is assessed under the
  same standards
• Standards of environment
• Standards of instruction/assistance
• Standards of assessing test performance
• Standards of comparison (norms)

22
What are norms?

• Norms are the standards for evaluating an
  individuals performance on an instrument (test).
• Who are norms based on?
• Anyone that can form a group. Examples
• Workers in a certain job.
• School Classrooms
• Geographic Location
• Culture/Ethnic Characteristics

23
Which norms should you use?

• Many tests give a variety of options.
• Choose the one that is most in common with
• Individual Characteristics
• Individual Goals (where will the individual be
  applying the attribute?)

24
Maintaining Standardization

• Putting the person in the norm group.
• Manual will give specific guidelines as to
• Environmental conditions in which the test should
  be taken.
• Specific instructions to be read/provided to the
  test-taker.
• Instructions for answering questions, giving
  assistance, scoring the instrument.

25
Standard Scores

• Standard scores are a means by which raw scores
  (such as number of right answers) can be compared
  to a norm group.
• Standard scores also provide a means for
  converting scores on two tests so that they may
  be compared to each other on a unit-to-unit
  basis.
• See Page 102 in text

26
Percentiles

• Percentiles indicate the proportion of scores
  that lie at or below a particular score.
• Example A score at the 75th percentile is as
  high or higher than 75/100 of scores recorded for
  the test.
• Percentiles and performance (traditional)
• 0-33 ileBelow Average
• 34-67ileAverage
• 68-99ileAbove Average

27
The Normal (Bell) Curve

• Characteristics
• Symmetrical
• Unimodalhas one high point mean (average
  score), median and mode are all the same in a
  normal curve.
• Infinitethe tails of the normal curve go on
  forever.
• Certain percentage of curve will fall within one
  two or three standard deviations of the mean.

28
Standard Deviations and Z-Scores

• Standard Deviationthe average difference
  (distance) between scores and the mean.
• In a normal curve, approximately 34 of scores
  will fall within one standard deviation above OR
  below the mean 14 within the next standard
  deviation, and 2 within the next standard
  deviation.
• The mean is at the 50th ile.
• A z-score represents the number of standard
  deviations an individuals score is away from the
  meanit is the most basic standard score.

29
Z-scores can be easily converted into raw scores
or rough percentiles.

• If you know the Z-score, the standard deviation,
  and the mean, you can figure a raw score
• Multiply the standard deviation by the z-score.
• If the z-score is negative, subtract this product
  from the mean score.
• If the z-score is positive, add the mean to this
  product.
• The result is the raw score.
• The z-score indicates how many standard
  deviations an individual is from the mean using
  the 34/14/2 rule, you can have a rough idea of
  the percentile at which this score lies on the
  normal curve.

30
Other Standard Scores

• T-score Mean of 50 standard deviation of 10.
• Deviation IQ score Mean of 100, standard
  deviation of 15.
• Weschler Subtest Score Mean of 10, standard
  deviation of 3.
• GATB/AGCT Score Mean of 100, standard deviation
  of 20

31
Converting Scores between Scales

• Convert the raw score to a z-score by determining
  the number of standard deviations away from the
  mean the raw score lies (positive or negative).
• Determine the standard deviation and mean of the
  scale to which you are converting.
• Multiply the z-score by the standard deviation of
  the scale to which you are converting.
• Add the mean of the scale to which you are
  converting if the score is positive.
• Subtract from the mean if the score is negative.

32
Example

• What is the equivalent GATB score for a Deviation
  IQ score of 130?
• Deviation IQ Mean 100 sd 15
• Score is 30 points above the mean
• 30/15 2.0 this is the z score.
• GATB Mean 100 sd 20.
• Multiply GATB sd by z score 2.0 x 20 40
• Because the raw score was above the mean, we add
  the mean to the product 100 40 140
• Deviation IQ of 130 is the same as a score of 140
  on the GATB scale.
• By the way, at what percentile would this score
  lie?
• Is it the same percentile on both scales?