Youve Been Shown: Now Its Your Turn to Ask

1
Youve Been Shown Now Its Your Turn to Ask!
Responding to Your Questions About
Testing
CLEAR 2004
Kansas City, Missouri
2
Credentialing and Exam Issues Sessions at CLEAR
2004

• Ten Year Review of CBT
• When Certification Replaces Licensure Show me
  Accreditation
• ADA Let Us Show You What Works
• Show Me What Globalization Means for
  Credentialing Programs
• Multiple Tracks to Licensure Show me the Way
• Test Piracy and Security in a Wired World

CLEAR 2004
Kansas City, Missouri
3
Credentialing and Exam Issues Sessions at CLEAR
2004

• Show Me the Validity
• Show Me How to Get Past MCQs New Opportunities
  in Measurement
• Show me an Evidential Approach to Assessment
  Design
• The Case of the Repeating Licensure Examination
  Candidate Psychometric, Legal, and Policy Issues

CLEAR 2004
Kansas City, Missouri
4
Moderator Chuck Friedman
Assistant Vice President, ACT
Panelists Carol Morrison Assistant Vice
President National Board of Medical
Examiners Elizabeth Witt Senior
Statistician American Board of Emergency
Medicine Rose McCallin Director of Office
of Examination Services State of
Colorado
CLEAR 2004
Kansas City, Missouri
5
Speakers You . . . the audience!!!
CLEAR 2004
Kansas City, Missouri
6
Validity
Reliability
Standard Administration
Item Writing
Item Analysis
Item Review
Passing Score
Scoring
Test Blueprint
Job Analysis
7
(No Transcript)
8
Scoring
Item Review
Reliability
Job Analysis
Item Writing
Validity
Standard Administration
Test Blueprint
Item Analysis
Passing Score
9
Should state licensing boards require continuing
education?
CLEAR 2004
Kansas City, Missouri
10
How do I find out if my test has been
compromised? If so, what should I do?
CLEAR 2004
Kansas City, Missouri
11
What should I do if I do or not do if I think a
candidate or candidates have cheated?
CLEAR 2004
Kansas City, Missouri
12
Test Compromise?

• Missing test book
• When, who?
• Statistical analyses
• Memorization of items
• Memorize and reconstruct items
• Study guides
• Internet postings
• Statistical analyses

13
Is a self assessment exam good enough for
continuing competency?
CLEAR 2004
Kansas City, Missouri
14
(No Transcript)
15
Four Components ofContinuous Certification

• Professional Standing
• Lifelong Learning and Self Assessment
• Assessment of Cognitive Expertise
• Assessment of Practice Performance

16
Considerations in Determining the Value of a
Self-Assessment

• How is the exam developed and scored?
• Can you be certain who took the exam?
• How high are the stakes for maintaining the
  credential?
• Does the self-assessment process foster learning?
• Is it easy to participate as part of ones normal
  professional activities?

17
Is performance testing useful in state licensure?
CLEAR 2004
Kansas City, Missouri
18
Performance Assessment

• Does performance assessment measure something
  different?
• Reliability, validity
• How do you score it?
• Cost, complexity
• Answer It depends!

19
What is your opinion about open book exams?
CLEAR 2004
Kansas City, Missouri
20
What should I do if a candidate receives an
incorrect score report?
CLEAR 2004
Kansas City, Missouri
21
How do you determine if a request for
accommodations is appropriate? If a test is
administered under special conditions, should the
score report be flagged?
CLEAR 2004
Kansas City, Missouri
22
Is Request Appropriate?

• Use expert consultant
• Level the playing field, dont tilt it too far
  the other way

23
To Flag or Not to Flag?

• Many testing companies have stopped flagging
• Some claim that flagging may lead to unintended
  consequences
• Standards for Educational and Psychological
  Testing (APA/AERA/NCME)
• Flag if do not have evidence of score
  comparability

24
Do we have to equate our tests? Is there one
best way?
ONE BEST WAY?
CLEAR 2004
Kansas City, Missouri
25
Reasons to Consider NOT Equating

• Small sample size
• Multidimensional exam
• Changing content outline

26
What If We Cant Equate?

• Create a detailed blueprint
• Write the best possible items
• Build forms that are closely parallel in both
  content and statistics
• Conduct frequent standard setting studies
• Include common items across forms study
  differences qualitatively

27
An examinee complains that he failed the test by
one point/question. What do you tell him?
CLEAR 2004
Kansas City, Missouri
28
Other Sources of Information

• CLEAR Exam Review
• Other CLEAR Publications
• ADA Handbook
• Development, Administration, Scoring and
  Reporting of Credentialing Exams
• CLEAR web site Frequently Asked Questions
• Joint Standards AERA, NCME, APA
• Professional psychometricians

CLEAR 2004
Kansas City, Missouri