ITEM WRITING

1
(No Transcript)
2
Welcome
3
TesT Writing Skills

• Chris Keegan, CST, MS
• Professor/Chair, Surgical Technology and Surgical
  Assisting Programs
• Vincennes University

4
Testing

• Are we developing good tests to measure student
  comprehension?
• What do our tests actually measure?
• How can we make our tests achieve what we want?

5
We put our Hearts In our Tests
6
The Reality

• Most times, we think our tests are simple,
  straight-forward measurements of the information
  which our students have retained.
• The students may think otherwise.

7
Item Writing

• Training provided through workshop
• All items linked to Core Curriculum
• Consistent style and format

8
What makes our Tests viable
9
Basics of Item Writing
10
Item components include a

• stem
• four options
• one key
• three distractors

11
Item Components

• Stem
• The statement or question to which candidates
  respond
• The stem can also include a chart, table,
  graphic, or scenario
• The stem should clearly present one problem or
  idea

12
Example Stems

• Direct question
• What is the primary purpose of a Deaver?
• Sentence completion
• The primary purpose of a Deaver is to
• Novice writers tend to write clearer direct
  question item types. If you are new to item
  writing, it is best to concentrate on that type.

13
Example Stems

• Never leave a blank in the middle or at the
  beginning of the stem
• ______ and ________ are self-retaining
  retractors.
• Which of the following are self-retaining
  retractors?
• Such items are more difficult for candidates to
  read.

14
Negatively worded item stems

• should be avoided
• try to rewrite as positively worded
• can cause measurement error when able candidates
  become confused

15
Negatively worded item stems

• Which of the following was NOT a Civil War
  battle?
• Normandy
• Bull Run
• Gettysburg
• Lookout Mountain

16
Negatively worded item stem rephrased as
positive

• Which of the following was a Civil War battle?
• Normandy
• Waterloo
• Gettysburg
• Trafalgar

17
Undirected Stems

• Undirected stems require the candidate to read
  all options to know what the item is asking
• Can the item be answered without looking at the
  options?

18
Undirected Stem -

• When Lincoln first signed the Emancipation
  Proclamation it
• allowed for the enlistment of black soldiers.
• was intended to keep other countries from
  recognizing the Confederacy.
• freed the slaves in the border states.
• absolved the former rebel states from the
  national debt of the Confederacy.

19
Undirected Stem - corrected

• When Lincoln first signed the Emancipation
  Proclamation his intent was to
• allow for the enlistment of black soldiers.
• keep other countries from recognizing the
  Confederacy.
• free the slaves in the border states.
• release the former rebel states from the national
  debt of the Confederacy.

20
Situational sets

• Longer scenarios can be constructed when
  supported by multiple items
• The key for each item must be independent of
  other items in the set
• Each item must still link to content and
  cognitive classifications from the detailed
  content outline

21
Some tests make you crawl out of your skin Stem
Skelton Options Muscle
22
Item Components

• Options
• one key
• the key is the best or most appropriate of the
  available options
• three distractors
• plausible, yet incorrect responses or completions

23
Things to doGeneral Rules for Writing
24
Write efficient, clear options

• Order options logically (short to long)
• Begin options with a letter (A, B, C, D) to avoid
  confusion with numerical answers
• Write options in similar lengths
• Avoid writing keys that are longer and more
  detailed than the distractors

25
Write efficient, clear options

• Write options that are independent
• First aid for a shock victim would include which
  of the following?
• A. lowering the body temperature
• B. rapidly cooling the skin
• C. elevating the legs
• D. orally rehydrating
• Since A and B are essentially the same a
  testwise candidate can eliminate them.

26
Write efficient, clear options

• Write options that are independent
• Avoid creating subsets of one option in other
  options

A. valves B. arterioles C. capillaries D.
arteries
27
Avoid mixing concepts in options
Write efficient, clear options

• A. apples
• B. oranges
• C. bananas
• D. hammers

A. carotid B. aorta C. coronary D. vena cava
28
Put as many words as possible into the stem
The psychometrician should recommend A. that the
committee write distractors of length similar to
the key. B. that the committee write mutually
exclusive distractors.

• The psychometrician should recommend that the
  committee write
• A. distractors of length similar to the key.
• B. mutually exclusive distractors.

29
Write in third-person style

• A surgeon is reviewing . . .
• An anesthesiologist is evaluating . . .
• A laboratory received . . .
• A surgical technologist scrubbed . . .
• Specifically avoid pronouns like you and your

30
Write distractors with care

• Item difficulty largely depends on the quality of
  the distractors.
• The finer distinctions candidates must make, the
  more difficult the item.
• When writing item stems, you should help
  candidates clearly understand.
• Distractors should be written with a ruthless
  attitude.

31
Write distractors you know some candidates will
select.

• Use common misconceptions
• Use the candidates language
• Use impressive-sounding and technical words
  (e.g., accurate, important) in the distractors
• Use scientific and stereotyped phrases, and
  verbal associations

32
Thingstoavoid
33
Do not use All of the above or None of the
above

• All of the above allows candidates to correctly
  respond based on partial information.
• None of the above is a poor option because
  candidates could almost always argue that it was
  correct.

34
Avoid Preaching and Teaching

• The examination is not the forum for telling
  candidates only about practices in your area.
  When sitting for a test candidates are not
  benefited by information that doesnt help answer
  the questions on a national level. The test
  should focus entirely on assessing candidate
  competence.

35
Regionalism is Taboo
36
Avoid irrelevant sources of difficulty

• Use a few short sentences rather than long
  complex sentences
• Use smaller words rather than polysyl-labic words
  to reduce reading level (Note this does not
  include medical terminology)
• Keep computations simple
• Items and options should be simply written
• Measure what is needed do NOT evaluate nebulous
  information

37
Avoid Definition Questions

• Correctly responding to such items may require
  little job-related mastery
• A correct response is more closely linked to
  instructional activities, which do not
  necessarily transfer to the job setting
• e.g., rather than ask candidates to define a term
  like cardiac arrest, include cardiac aarrest in
  the stem so candidates must interpret the term as
  a part of the larger problem (i.e. death of a
  patient)

38
Avoid Convergence Clues

• When options repeat the same elements, but in
  different combinations, the key is often the
  option with the most frequently used elements

39
Avoid Convergence Clues

• A. aorta and carotid
• B. aorta and coronaries
• C. subclavian and carotid
• D. femoral and iliac

40
Convergence is easy to fix

• A. aorta and carotid
• B. iliac and carotid
• C. aorta and coronaries
• D. coronaries and iliac

• ?

41
Novice writers often provide irrelevant clues by
writing

• options having the same meaning. Therefore, both
  must be incorrect.
• options that are mutually exclusive (e.g., dose
  is correct dose is incorrect).
• Candidates will be drawn to choosing among the
  two parallel options. Probability of guessing
  increases to 50 measurement error inflates.

42
Novice writers provide irrelevant clues by

• placing similar phrases in the stem and key, even
  including identical words
• writing keys better than distractors
• writing the key in more technical, detailed
  language
• using modifiers only associated with
• true statements (e.g., may, sometimes, usually)
  for keys
• false statements (e.g., never, all, none, always)
  for distractors

43
Novice writers provide irrelevant clues by

• Using Mom and apple pie correct answers
• Testwise candidates will be drawn to answers
  like
• within legal guidelines.
• according to established safety procedures.
• If an option simply reads like something
  impossible to disagree with it is probably the key

44
Cognitive levels

• Recall
• Application
• Analysis
• Cognitive levels are designated because we
  recognize that varying dimensions of the job
  require varying levels of cognition. By linking
  items to cognitive levels, a test better
  represents the job.

45
Cognitive Level Examples

• Recall
• ½ x 1/3 ?
• How many pounds are in a kilogram?
• Only requires memory, but may be difficult.
• Usually involves single concepts.

46
Cognitive levels

• Application items
• use interpretation, classification, translation,
  or recognition of elements and relationships
• Any item involving manipulations of formulas, no
  matter how simple, are application level
• Items using graphics or data tables will be at
  least at the application level
• have keys that depend on the situation presented
  in the stem
• If the key would be correct in any situation,
  then the item is probably just a dressed up
  recall item

47
Cognitive Level Examples

• Application
• 100 C ? F
• On the medication label shown above, what is the
  drugs generic name?
• Goes beyond straight memory. Can involve 2 or 3
  simple, related concepts. In the math example,
  multiplication tables and addition are required.

48
Cognitive levels

• Analysis items
• use information synthesis, problem solving, and
  evaluation of the best response
• require candidates to find the problem from clues
  and act toward resolution

49
Cognitive Level Examples

• Analysis
• - A patient presents to the ER with a ruptured
  spleen. He has no identification of any kind and
  is alone. Which of the following is true
  regarding his operative consent?
• Involves multiple concepts that must be put
  together in a meaningful way. In the math
  example, square root, exponents, multiplication,
  and division must be performed in the correct
  order.

50
Tips for writing items to test higher cognitive
levels

• Use job-related situations that require
  consideration of multiple issues to arrive at a
  solution
• Do not explicitly identify the problem when
  prompting trouble-shooting

51
Tips for writing items to test higher cognitive
levels

• Give clues to the problem and ask for the best
  course of action, or what should be done first
• Options typically describe actions, not things

52
Are you testwise?
You are testwise if you can select the key based
on clues given in the item without knowing
the content.
53
1. Trassign normally occurs under which of the
following conditions?A. when dissles
frullB. when lusp trasses the vomC. when the
belgo lisks tariousD. when the viskal flans, if
the viskal is zortil
If you are testwise,you can find the key.

• Testwise candidates will pick D as the key it is
  the only option to specify a condition and the
  stem directs candidates to look for a condition.

54
2. The fribbled breg will ninter best with
anA. mors.B. ignu.C. derst.D. sortar.
If you are testwise, you can find the key.

• Testwise candidates would select B it is the
  only option that grammatically flows from the
  stem.

55
3. Why does the sigla frequently overfesk the
trelsum?A. All siglas are mellious.B. Siglas
are always votial.C. The trelsum is usually
tarious.D. No trelsa are directly feskable.
If you are testwise, you can find the key.

• Testwise candidates can identify C as the key
  because all the other options contain words
  (i.e.. all, always, no) associated with false
  statements.

56
4. What probable causes are indicated when doss
occurs in a compots?A. The sabs foped and the
doths tinzed.B. The kredges roted with the
rots.C. The rakogs were not accepted in the
sluth.D. The polats were thenced in the sluth.
If you are testwise,you can find the key.

• This one is relatively tricky, yet some testwise
  candidates may be able to deduce that A is the
  only option with multiple causes (sabs and doths).

57
5. What is the primary purpose of the cluss in
frumpaling?A. remove cluss-prangsB. patch
tremailsC. loosen cloughsD. repair plumots
If you are testwise,you can find the key.

• Someone who knows nothing about the content can
  still pick A as the key because its the only
  option to include a word (cluss) from the stem.

58
6. The nintering function of the ignu is most
effectively performed in connection with theA.
arazma tol.B. fribbled breg.C. groshing
statol.D. frallied stantels.
If you are testwise,you can find the key.

• Testwise candidates will remember that ignus and
  fribbled bregs were linked in the second item and
  thus select B here. This is an example of cueing.

59
Preparing to Write

• You must link your items to the prescribed
• part of the detailed content outline
• cognitive level
• You will write multiple-choice items with
  four-options

60
Assignments

• Items must be written to content outline
• Make sure to note key, content area, and
  cognitive level
• Please write legible items and avoid
  abbreviations

61
Our thanks goes to

• Mr. Jeff Kelley of Applied Measurement
  Professionals, Inc
• Body Worlds which will be exhibiting in Chicago
  February 4 September 5, 2005
• All of you

62
(No Transcript)
63
(No Transcript)