Basic Testing Issues

1
Basic Testing Issues
2007 Annual Conference

• Ron Rodgers
• CTS/Employment Research Institute

2
Session Agenda

• A. Clarify test development Standards
• Summarize test development process
• Write validate multiple choice items
• Respond to your questions

3
APA Standards (1954-1999)

• APA Standards used by US Supreme Court to
  validate fair, valid examinations
• Test must reflect job qualifications
• Credentialing identifies qualified practitioners
  for public, employers
• Panels of experts should help specify
  certification performance standards
• Standards for Educational and Psychological
  Testing, American Psychological Association,
  American Educational Research Association,
  National Council on Measurement in Education

4
APA Standards(1954-1999)

• Candidates who fail receive report w/information
  about weak areas
• Credentialing must depend on the knowledge and
  skills needed for acceptable professional
  performance
• Criteria should not be adjusted to regulate the
  number or proportion of persons who pass the test

5
Define Exam Objectives

• Create SME panel to represent experienced
  professionals in field
• Identify references initial specs
• Conduct job analysis
• Link test specifications to job tasks

6
Write Validate Questions

• Select item writing model
• Draft item bank to meet test specs
• Document each Q to references
• Validate/edit Qs with SME panel

7
Score Report Results

• Assemble test forms
• Set passing criteria
• Review item performance statistics
• Equate test forms
• Adjust scoring if needed

8
Provide Candidate Info Tech Support

• Prepare candidate materials
• Monitor test item performance
• Revalidate non-performing Qs
• Amend items/test forms as needed

9
Maintain Test Security

• Limit access to secure materials
• Use non-disclosure agreements
• Emphasize security to candidates
• Define consequences of cheating

10
Non-Disclosure Agreement

• Each Subject Matter Expert (SME) acknowledges
  that all work performed under this Agreement for
  and on behalf of (program sponsor) is
  confidential and that no disclosure of any
  aspects of test data, information, materials and
  questions is permitted. The SME also agrees to
  adhere to all test security procedures and
  requirements specified by __________.

11
Responding to Your Questions

• Writing good test questions
• Bruce Anderson, PES
• Identifying your concerns
• Bruce Anderson, PES
• Ron Rodgers, CTS
• Finding sources of assistance

12
Speaker Contact Information

• Ron Rodgers
• Continental Testing Service
• Employment Research Institute
• 809 Ridge Road, Suite 201
• Wilmette, IL 60091
• Phone [phone]
• Email rrodgers_at_Qmail.com

13
Writing Good Multiple-Choice Items
2007 Annual Conference

• Bruce Anderson
• Professional Examination Service

14
Agenda

• Overview of the examination development process
• Review of item-writing principles
• Filling out the item-writing form
• Review and development of items

15
Developing the Certification Examination

• Conduct a job analysis/role delineation.
• Generate test specifications.
• Write items.
• Validate/Improve items.
• Enter items into item bank.
• Assemble draft examinations.
• Review and revise examination forms.

16
Developing the Certification Examination (contd.)

• Conduct a passing point study.
• Review item statistics.
• Equate examination forms.
• Score and report examination results.

17
What are the chances of an uninformed candidate
selecting the correct answer to your item?

• 1 in 4, and thats if you write a good item!!

18
Guessing the Correct Answer

• For a multiple-choice item with four possible
  responses
• 3 good distracters a 25 chance
• 1 BAD distracter a 33 chance
• 2 BAD distracters a 50 chance
• 3 BAD distracters a 100 chance
• of an uninformed candidate guessing the
  correct answer.

19
The Effects of Guessing on the Value of the
Certification

• The greater the ability of the uninformed
  candidate to guess correctly, the less valid and
  reliable the examination is.
• The less valid and reliable the examination is,
  the less value is placed on the certification.

20
The Critical Objective when Writing Good
Multiple-Choice Items

• Reducing the ability of uninformed candidates to
  guess correctly by providing NO CLUES to the
  correct answer.

21
Where are the clues which help candidates to
guess correctly?

• The type of format chosen for the question
• The format of the question asked (STEM)
• The way the correct answer (KEY) is written
• The way the incorrect answers (DISTRACTORS) are
  written

22
The Format of the Stem

• DIRECT QUESTION
• Which of the following scientists was responsible
  for the development of the first polio vaccine?
• or
• INCOMPLETE STATEMENT
• The first polio vaccine was developed by
• Jonas Salk
• Louis Pasteur
• Edward Jenner
• Robert Koch

23
Correct vs. Best Answer Stems

• What is the capital of Kentucky?
• Frankfort
• Lexington
• Louisville
• Ashland

• The major goal of raising the legal drinking age
  is to
• Reduce driving fatalities and injuries.
• Reduce liquor consumption.
• Discourage the habit of drinking by adolescents.
• Reduce roadside litter.

24
Guidelines for Constructing the Stem

• Clarity and Directness Is the requirement of the
  item clear without reference to the choices?
• Sufficiency Is enough information provided to
  answer the question?
• Focus Is the item focused on a single problem
  which assesses a meaningful fact or concept?

25
EXAMPLE Clarity? Sufficiency? Focus?

• The 2000 U.S. Presidential Election was decided
  by
• 1. fewer than 1000 votes
• 2. the Electoral College
• 3. the Supreme Court
• 4. December 21, 2000

26
Guidelines for Constructing the Stem (contd)

• Avoid negative stems.
• Provide just enough information.
• Do not use a word in the stem which only appears
  in the choice which is the key.
• Do not provide gratuitous instruction.
• Provide most information in the stem so that the
  options do not have repetition of words.

27
EXAMPLE Negatives are confusing.

• Which of the following authors is not a major
  American novelist?
• 1. George Orwell
• 2. William Faulkner
• 3. John Steinbeck
• 4. Ernest Hemmingway

28
EXAMPLE Improved

• Which of the following authors is a major British
  novelist?
• 1. George Orwell
• 2. William Faulkner
• 3. John Steinbeck
• 4. Ernest Hemmingway

29
EXAMPLE Gratuitous instruction

• Poor
• In 1850, Adolphe Chatin, the professor of
  pharmacy in Paris, believed that goiter resulted
  from an inadequate amount of iodine in the diet.
  The thyroid is part of which body system?
• Endocrine
• Nervous
• Musculoskeletal
• Cardiovascular

• Better
• The thyroid is part of which body system?
• 1. Endocrine
• 2. Nervous
• 3. Musculoskeletal
• 4. Cardiovascular

30
Guidelines for the Correct Answer (KEY)

• Is the intended key unambiguously correct and
  recognized as such by the experts?
• Is the key parallel to the other choices (e.g.,
  not shorter or longer, or more technical, or more
  detailed)?
• Is there ONLY ONE clearly correct or best answer?

31
Guidelines for Distracters

• SOURCES OF DISTRACTORS
• Common misconceptions
• Common job performance errors
• Statements which are true but which do not
  satisfy the requirements of the question
• Statements which are either too broad or too
  narrow for the requirements of the question
• Carefully worded incorrect statements which may
  sound plausible to the uninformed

32
Guidelines for The Choices

• Does each choice follow logically and
  grammatically from the stem?
• Are the choices parallel in
• length?
• grammatical structure?
• terminology?
• content?
• level of detail?

33
EXAMPLES Parallelism

• An American city noted for its production of
  automobiles is
• 1. Detroit
• 2. Ohio
• 3. Indiana
• 4. Philadelphia

• An American city noted for its production of
  automobiles is
• Detroit
• Cincinnati
• Indianapolis
• Philadelphia

34
EXAMPLES Parallelism, length.

• Serious communicable diseases that can cause
  developmental disability to the offspring of a
  pregnant mother are
• 1. rubella and chicken pox
• 2. cancer
• 3. rheumatism and arthritis
• 4. sinus infection

• A serious communicable disease that can cause
  developmental disability to the offspring of a
  pregnant mother is
• rubella
• cancer
• rheumatism
• sinus infection

35
Guidelines for Choices (contd.)

• Do not repeat the same word in each answer
  choice include the word once in the stem. As
  much time as possible is to be spent figuring out
  the answer rather than reading.
• Make the choices mutually exclusive.
• Do not use all of the above or none of the
  above as choices. The latter, for example, does
  not test whether a candidate knows the correct
  answer.

36
EXAMPLES Overlap

• The right to vote in the United States is granted
  to individuals of what age?
• 1. 16
• 2. 17
• 3. 18
• 4. 19

• The right to vote in the United States is granted
  to individuals beginning at what age?
• 1. 16
• 2. 17
• 3. 18
• 4. 19

37
General Considerations

• Is it RELEVANT to professional practice?
• Is current BEST practice in the field tested?
• Does it test an important concept of piece of
  knowledge?
• Is it appropriate for entry-level?
• Is the answer clear to a prepared candidate (not
  a trick question)?

38
General Considerations

• Does it avoid controversial issues, e.g.,
  religious, political, philosophical?
• Are the context, setting, and content equally
  appropriate and familiar to all segments of the
  candidate population?
• Is the terminology equally appropriate and
  familiar to all segments of the candidate
  population (e.g., international candidates)?
• Is it free of bias re persons or groups?

39
EXAMPLE Sensitive Issue

• Of the following profile analysis groups, the one
  most likely to have alcohol-related problems is
• 1. residents of rural areas in the South
• 2. widowed males over 60 years of age
• 3. women who are married and are wine drinkers
• 4. men with no religious affiliation

40
Other Sensitivity Issues

• Avoid humor in test questions.
• Avoid any subject that appears to be advancing a
  political or business agenda.
• Avoid topics that could cause undue stress to the
  candidates, e.g., death, disease, war, brutality.
• Avoid any stereotypes.

41
Rules of English

• Dont use no double negatives.
• Make each pronoun agree with their antecedent.
• Join clauses good, like a conjunction should.
• Watch them participles when dangling.
• About them sentence fragments.
• Verbs has to agree with their subject.
• Just between you and I, case is important too.
• Dont write run-on sentences they are hard to
  read.
• Dont use commas, that are not necessary.
• Dont misuse apostrophes its wrong.

42
THE I GNU TEST
43
1. The purpose of the cluss in furmpaling is to
remove

• 1. cluss-prags
• 2. tremalis
• 3. cloughs
• 4. plumots

44
2. Trassig is true when

• 1. lusp trasses the vom
• 2. the viskal flans, if the viskal is donwill or
  zortil
• 3. the belgo frulls
• 4. dissles lisk easily

45
3. The sigla frequently overfesks the treslum
because

• 1. all siglas are mellious
• 2. siglas are always votial
• 3. the treslum is usually tarious
• 4. no trelsa are feskable

46
4. The fribbled breg will minter best with an

• 1. derst
• 2. morst
• 3. sortar
• 4. ignu

47
5. The mintering function of the ignu is most
effectively carried out in connection with

• a razma tol
• the groshing stantol
• the fribbled breg
• a frally sush

48
Problem Items
49
Problem Items

• Which of the following persons was President of
  the United States when the Civil Rights Act of
  1964 was enacted?
• A. Lester Pearson
• B. Anthony Eden
• C. Jefferson Davis
• D. Lyndon Johnson

50
Problem Items

• Who orchestrated the December 7, 1941 attack on
  Pearl Harbor?
• A. Isoroku Yamamoto
• B. Chester Nimitz
• C. Bernard Montgomery
• D. Seiji Ozawa

51
Problem Items

• Which Swedish opera singer was known for her role
  in Tristan und Isolde?
• A. Liv Ullman
• B. Birgit Nilsson
• C. Jussi Bjorling
• D. Ingmar Bergman

52
Contact Information

• Bruce R. Anderson
• Senior Program Director
• Professional Examination Service
• 475 Riverside Drive
• New York, NY 10115
• Phone [phone]
• Fax  [phone]
• anderson_at_proexam.org