Recovering from the Unexpected

1
Recovering from the Unexpected
Presented at The 22nd Annual CLEAR
Conference September 12 14, 2002
Panel Ronald C. Rodgers, PhD Consulting
Psychometrician Continental Testing
Services President, Employment Research
Institute Randolph P. Reaves, JD Executive
Officer and General Counsel Association of State
and Provincial Psychology Boards Elizabeth A.
Witt, PhD Senior Measurement Analyst Promissor
(formally CATASI)
Moderator I. Leon Smith, PhD President CEO PES
2
Scenario Summary
The unexpected arrived in the form of an email
to the licensure program sponsor to look at the
website of a test preparation company. A review
of the site revealed a number of pages containing
the full text of items presumably recalled from
the examination by some candidates. A check of
the text of the items revealed that approximately
5 of the items were identical to those on the
four current forms of the examination being
delivered through the computer delivery vendor.
Most of the exposed items also appeared on the
four new forms already at the computer delivery
vendor to be deployed shortly as replacements for
the four current forms. Each form contains
operational (scored) items, and several pre-test
(unscored) blocks. All forms are pre-equated.
To ensure the timely implementation of the four
new forms, any replacement of items must be
limited to those items already on the compute
delivery vendors system. Since the test
preparation website has been operational for
several years, the magnitude of exposure probably
went beyond these few items. In addition,
approximately 70 of the items on the four new
forms overlap with the current forms being
administered.
3
Objective

• The licensure program sponsor wants to accomplish
  the following goals
• Ensure the continuing validity of examination
  scores and the public protection function of
  licensure examination forms
• Close down the test preparation website that is
  posting the exposed items
• Have the Internet host provider terminate the
  service to the test preparation website
• What to do? And what are the issues?