Development of Study Guides and Exams

1
Development of Study Guides and Exams A
Colorado Perspective

• Sandra K. McDonald
• Betsy Buffington
• Colorado State University

2
Study GuideDevelopment
Exam Development
Recruit Item Writing Committee
Develop Draft Test Items
Select Guide
Recruit Study Guide Workgroup
Validate Draft Test Items
Orientation Workshop
Develop Guide Outline/Blueprint
Finalize Test Items
Conduct Job Analysis
Develop Study Guide
Enter Test Items Into Database
Validate Existing Questions
3
Select Guide

• Pass/fail rates of category
• Number of applicators testing/year
• Rotate between Agricultural, Ornamental, and
  Structural

Select Guide
4
Recruit Study Guide Workgroup

• CDA requests professional organizations to submit
  volunteer(s)
• Individuals may also volunteer themselves
• All levels of experience
• Certified Operator
• Mixes, loads, or applies any pesticide, including
  RUPs, under the supervision of a QS
• Qualified Supervisor
• Without supervision, evaluates pest problems,
  recommends pest controls using pesticides or
  devices, mixes, or loads, or applies any
  pesticide, sells application services or
  operates devices
• Supervises in any of these functions
• Trains Technicians

Recruit Study Guide Workgroup
5
Develop Guide Outline/Blueprint

• One day question and answer session
• Discuss what information to include
• What is important for entry-level pesticide
  application?
• Equipment types
• Pesticides used
• Refer general information to Core
• Develop outline and order of chapters
• Layout mirrors exam categories
• Job analysis

Develop Guide Outline/Blueprint
6
Guide Blueprint
7
Develop Study Guide - Writing

• Glean information from other study guides
• Tailor to Colorado
• Workgroup participants provide additional
  resources
• Emails
• Other references

Develop Study Guide
8
Writing Study Guides
9
Develop Study Guide Review

• Send copies to CSU staff, CDA staff and Industry
• Workgroup participants plus other key identified
  individuals
• Highlight particular chapters to review
• Ensures all chapters will be reviewed in-depth
• Reduces amount of time spent reviewing guide
• CEPEP and CDA staff review entire guide
• Incorporate comments when guides are received
• Second review

10
Develop Study Guide Publish
11
Study GuideDevelopment
Exam Development
Recruit Item Writing Committee
Develop Draft Test Items
Select Guide
Recruit Study Guide Workgroup
Validate Draft Test Items
Orientation Workshop
Develop Guide Outline/Blueprint
Finalize Test Items
Conduct Job Analysis
Develop Study Guide
Enter Test Items Into Database
Validate Existing Questions
12
Recruit Item Writing Advisory Committee

• CDA requests each professional organization to
  submit a list of volunteers
• Individuals can also volunteer
• Committee participants
• Must be licensed as CO or QS in category
• Various levels of experience
• Security Agreement
• All test items become sole propertyof CDA

Recruit Item Writing Committee
13
Security Agreement
14
Orientation Workshop

• Teach participants basic skills necessary to
  write high quality exam items
• Discuss pros and cons of each question type
• True/False
• Multiple Choice
• Discuss dos and donts of writing good items

Orientation Workshop
15
Conduct Job Analysis

• Determine topics to be included
• i.e., applicator safety, laws and regulations,
  calibration, pest ID, etc.
• Determine the percentage of questions that should
  come from each topic
• CO and QS exams may be weighted differently
• Results of job analysis serve as blueprint for
  exam

Conduct Job Analysis
16
Example Job Analysis
17
Validate Existing Questions

• Committee edits each item for
• Content
• Format
• Style
• Grammar
• Absence of potentially inflammatory or biased
  language
• CO questions must be covered in study guide
• QS questions may be in study guide or from other
  sources
• Item(s) may be
• Accepted
• Reworked and accepted
• Rejected
• Referred to another item workgroup

Validate Existing Questions
18
Validating Existing Questions
19
Develop Draft Test Items

• Participants asked to submit questions
• Usually 5 or more questions/participant
• May assign questions to fulfill test blueprint

Develop Draft Test Items
20
Assign Draft Test Items

• Subject Pesticide Label Labeling, A. Read
  understand label requirements
• Need at least 6 CO mc and 2 QS mc (pgs. 57-67)
• Subject Pesticide Label Labeling, B.
  Understand that the label is considered a legal
  document
• Need at least 1 QS mc (pgs. 57-67)
• Subject Pesticide Label Labeling, C.
  Understand label terminology
• Need at least 1 QS mc (pgs. 57-67)

21
Item Submission Form
22
Validate Draft Test Items

• Items are reviewed and critiqued by the full Item
  Writing Workgroup
• Item(s) may be
• Accepted
• Reworked and accepted
• Rejected
• If accepted, the item becomes property of CDA

Validate Draft Test Items
23
Finalize Test Items

• Make sure minimum requirements are met
• Make sure 75 of QS questions are answered in
  study guide
• Check to see if true/false ratio is 50
• Ensure we have produced a minimum of 150 of
  total items needed

Finalize Test Items
24
Enter Finalized Test Items into Database

• Enter items into computerized item banking system

Enter Test Items Into Database
25
What We Have Learned

• Mix of experience levels are best
• Individuals contribute differently
• Industry buy-in results in
• Industry ownership of system
• Support for future development of guides/exams
• Cheap labor
• Relationship between CEPEP and CDA key to success

26
Training Blueprint