Developing Job-Related Test Materials Using the

1

• Developing Job-Related Test Materials Using the
• CATS SYSTEM
• CODESP Automated Test System

2
WHAT IS CODESP?Cooperative Organization for the
Development of Employee Selection Procedures

• A nonprofit consortium of over 700 public school
  districts, community colleges and charter schools
  which was formed in 1973 to combine resources to
  develop classified test materials
• Products include Written Test Items, Interview
  Questions, Supplemental Application Forms,
  Performance Exams, Sample Tests (including
  Instructional Aide) and Writing Exercises
• Includes over 300 small districts who have
  free membership through their County Office of
  Education because their ADA is less than 1,000.

3
How Can CODESP Help?

• CODESPS Item Bank, soon to be called (CATS),
  contains over 17,000 items covering most job
  classes found in public school districts
• A job-related test can be easily developed from
  the items CODESP provides
• Multiple types of assessment tests are available

4
How Can You Help?

• You can be instrumental in creating
• job-related exams
• By
• Conducting SME item reviews and
• Job Analysis
• And
• Notifying CODESP regarding SME Review and Job
  Analysis results to update test items

5
If Insufficient Attention is Given to the
Districts Employee Selection Process

• Employment-related lawsuits may occur a single
  legal challenge by an aggrieved applicant can
  cost thousands of dollars in government fines and
  tort liability
• Wasted recruitment time money when positions
  must be reopened
• Wasted salary and benefits to employees who are
  poor matches or who only stay a limited time
• Wasted training dollars and time
• Poor service to customers
• Reduced morale among current employees who must
  work with non-qualified co-workers

6
Ideal Selection Results In

• Job-Related/Valid Selection Devices/Tests
• Accurately Measured Candidate Ability
• Increased Quality of Candidates Selected
• Successful Organizations

7
Content Validity

• Referred to in the Uniform Guidelines and many
  court cases
• The content of the test must represent important
  job-related behaviors
• Test items should be relevant to and measure
  directly important requirements and
  qualifications for the job (KSAs)
• Also known as job-relatedness or face validity
• Achieved through job analysis

8
Job-Relatedness

• To achieve content validity each test item must
  be related to the job
• If not, the validity of the selection procedure
  is jeopardized
• Validity ? legally defensible

9
Court Case

• Griggs v. Duke Power
• Outlined the need for conducting an analysis
  of the job for which a selection procedure has
  been developed
• Artificial barriers which are not job-related
  should be removed from the selection process
• Tests should assess minimum requirements needed
  at the time of hire
• Tests should measure how the person will
  potentially perform on the job, not the person in
  the abstract

10
The Problem

• It is difficult for HR professionals to be
  experts and make judgments about the job
  relatedness of test items in all job subject
  areas

11
The Solution

• Gather information about the job and test items
  from SMEs
• Subject
• Matter
• Experts

12
How Do We Find SMEs?

• Use existing knowledge of district personnel
• Contact schools/department heads for suggestions
• Build lasting, positive partnerships with
  departments and SMEs for future projects
• If you are unable to find SMEs, contact CODESP,
  or other member districts directly

13
Steps To Take When Working With SMEs To Develop A
Selection Test

• Conduct a job analysis or check to see if major
  changes have occurred since the last job analysis
  (w/SME)
• Review current Job Description
• 3) Obtain potential test items (create them
  yourself or obtain them from CODESP)
• 4) Review potential test items (w/SME)
• Select the best items for the test
• Create and Print test online using CATS

14
Purpose Of Job Analysis

• Ensures that your current job description is
  correct and up to date
• Identify the knowledge, skills, and abilities
  (KSAs) required at the time of hire to perform
  the job successfully
• Is required to increase the validity of the test
  items

15
Job Analysis Process

• CODESPs new Job Analysis Tools
• Job Analysis Questionnaire
• Job Description Builder
• Products will be available Summer 07
• See Job Analysis Basics Training under Resources
  / Training Presentations
• See Resources / Job Description Builder and
  Questionnaire

16
Item and Job-Level Match

• It is important to determine whether the
  knowledge measured by an item is essential for
  the successful job performance of
• entry level employees
• intermediate level employees
• high level employees
• Items should only test knowledge that is
  essential at the time of hire

17
Item Writing Guide

• The following are a series of hints and
  suggestions to help your Subject Matter Experts
  construct or review test questions
• Write out the full title when describing objects
  and procedures. Only use acronyms when you are
  positive that your audience knows exactly what
  you are describing. When in doubt, spell it out.
• Clearly indicate the correct answer for each
  question. There should only be one best
  answer.

18

• Stay away from slang because these words can be
  open to different interpretations
• Try to avoid distracters that include words like
  always and never because they are generally
  false statements as well as words like some and
  usually because they are generally true
  statements
• Ask questions that have definite answers so that
  most experts will agree on the correct response
• Try to have a balance of easy, moderate, and hard
  questions. All disciplines need to be balanced
  with respect to difficultly level.
• Dont ask for details that only individuals who
  have worked in that particular department would
  know.
• Watch for hints in the stems (e.g., synonyms).
• Do not create questions that have negative stems
  (Which one of the following is not), unless
  you are addressing a certain action that all
  employees need to should be never done.
• Questions that have complex answers are less
  efficient and provide less reliable results.

19
PROBLEM ITEMS

• Not applicable to job/field
• Terminology too complex
• Outdated
• Incorrectly keyed
• Question Unclear
• Choice Unclear
• Other choices are also correct
• Other choices are too obvious
• Choices are not parallel

20
SME Test Item Security

• Use A SME Background/Security information form
  (found under Resources / Exam Planning) and keep
  it in the test file
• Make sure that CODESP test materials are stored
  in a secure location

21
CODESPCATS SYSTEM

• CODESP Automated Test System

22
Getting Started To get started, launch an
Internet Explorer session and enter the following
web address http//www.codesp.com. Then, press
the Enter key on your keyboard.

• This section will cover the following topics
• Website Navigation
• Registering for Services
• Logging In
• Logging Out

23
(No Transcript)
24

• Navigating the CODESP Website
• The CODESP banner and the navigation menu make
  information readily accessible, making website
  navigation a breeze.
• Some information is only accessible to registered
  members who have logged in.

25
Helpful Hints

• Because this website makes use of pop-ups, it is
  important to disable any pop-up blocker software
  that may be installed on your machine. You may
  be able to disable pop-ups on our website without
  disabling pop-up blocker software entirely
  depending on the software installed.
• Graphical icons are often used to simplify
  website interface. Simply, hover your mouse over
  the icon to view the functionality.

26
you
Member Name
Navigating the Website Your member start page
provides you with a customized view of CODESP
training events, your class registrations,
newsletters, announcements and your test material
request status.
27
Currently Registered Members

• The next several slides provide instructions on
  registering for CODESP
• If you are already registered, skip to Slide 55

28

• Registering for Services
• Becoming a CODESP member is easy and offers many
  benefits. With CATs CODESP members have access
  to
• Test Materials
• Tutorials
• Job Analysis Questionnaire
• Job Description Builder
• Training
• HR Resources
• Learn more about the different memberships CODESP
  offers by clicking on the Membership tab found on
  the toolbar.

29

• Registering for Services Continued
• To become a member of CODESP, click on Register
  Now and complete the following requirements
• Application for Services
• Security Agreement
• Membership Agreement   
• Purchase order or check for the full amount of
  the membership fee
• Not needed by districts and Charter Schools with
  an ADA of 1,000 or less if County Office is a
  member

30
New Members Register on the CODESP Home Page
Once you are registered you will receive a Member
Log-in Name and Password so that you can access
the secure areas of the website.
31
To begin the registration process, select the
desired membership. Mid-Year is only available
from Jan June and requires a commitment for a
full-year membership the following year.
Districts and Charter Schools with an ADA less
than 1,000 do not need to complete these
requirements, as long as their County Office is a
CODESP member. They only need to complete an
application and security agreement. Their
membership is free.
32

• The CODESP Registration Wizard guides you through
  each step of the registration process
• Step 1 Application for Services
• Select your County and District from drop down
  menu.
• Verify the address.
• If address is incorrect, make the appropriate
  changes.
• Complete the rest of the form, including a
  primary and secondary contact person(s).
• Click Next to move to the following screen.

33
(No Transcript)
34

• If your district does not appear on the drop down
  menu, click What if my district is not listed
  above?
• Fill out the form as seen in the next figure and
  click the submit button. You will be contacted by
  a CODESP representative in order to verify your
  request. Once CODESP has added your district, you
  will be able to complete the registration process.

35
(No Transcript)
36

• Step 2 Membership Agreement
• Read the Membership Agreement thoroughly.
• To comply, click I agree with the terms and
  conditions as stated above.
• Click Next to move to the following screen.
• Note You will not be able to continue
  registering unless you accept the terms.

37
(No Transcript)
38

• Step 3 Security Agreement
• Similarly to Step 2, read the Security Agreement
  thoroughly.
• To comply, click I agree with the terms and
  conditions as stated above.
• Click Next to move to the following screen.
• Note You will not be able to continue
  registering unless you accept the terms.

39
(No Transcript)
40

• Step 4 Registration Information Review
• Verify that the registration information provided
  is correct.
• Click Edit to revise the information.
• Click Next to move to complete the CODESP
  Registration Wizard.

41
(No Transcript)
42

• Step 5 Print, Sign and Fax Agreement
• CODESP is legally obligated to keep a signed
  contract on file for every CODESP member.
• Please print all forms.
• Sign them.
• Fax or mail them to the CODESP office
• 20422 Beach Blvd. Suite 310
• Huntington Beach, CA 92648
• Fax [phone]

43
(No Transcript)
44
The Registration Wizard ends at Step 5, but
remember, in addition to sending the signed
agreements, dont forget to send your membership
payment. Checks and purchase orders are
accepted. Forward an Invoice to your Business
Office. An invoice is available in the Document
Library. Once CODESP has received the membership
payment in full and the signed agreements, CODESP
will notify the primary contact and your district
will be given a District Administrator
account. The District Administrator will be
assigned a login name and password. To learn
more about the District Administrator role and
responsibilities, see section titled, District
Administrator Account.
45
Logging In If your account has already been
established, click Log In. Otherwise, contact
your District Administrator in order to set up an
account.
46
Logging In Continued First time users must agree
to the terms of the security agreement even if
one has been completed by other district users
47
Logging Out Throughout the course of the day,
there will be times in which you will not need
access to the systemlunch time, breaks,
meetings, etc. In these instances, you will need
to exit the system. In order to maintain the
highest level of security, end your session by
clicking on Log Out. To Keep the Site Secure
Always Log-Out When You are Not at Your
Workstation
48
My Profile Your profile can be viewed by
clicking My Profile, located on the navigation
bar. You will notice certain fields are
editable, while others are for viewing purposes
only. For security reasons, a person cannot
change their own Status or Security Role. This
data can only be changed by another District
Administrator. Never use another persons log-in
name and password
49
(No Transcript)
50

• Membership
• District Directory
• Your district will benefit from networking with
  other districts. The District Directory serves
  as a CODESP phone book and lists all the
  districts that are CODESP members.
• To access the District Directory, click District
  Directory located on the navigation bar.
• The searching tools provided make it easy to find
  a districts contact information. You can search
  by text or by county.
• Searching by Text
• Click Text as it is highlighted in the following
  figure.
• Enter keyword(s) in the Search For field.
• Click drop down menu to view the searchable
  fields.
• Highlight the field you wish to perform a search
  on.
• Click Search Now

51
(No Transcript)
52

• Search by County
• Click County as it is highlighted in the figure
  below.
• Click drop down menu to view list of counties.
• Highlight the county you wish to find.
• Click Search Now.

By default, your search results will be displayed
5 districts per page, but conveniently, you can
change the view to display 10 districts per page,
or if you prefer, you can select Show All to view
all the CODESP members.
53

• District Directory Continued
• The results page displays information about the
  district, including
• District Name
• Address
• Web Address (if information was provided)
• The icons on the right-hand side serve a purpose
  as well. To find out their functionality, simply
  hover your mouse over each icon and a description
  will appear.
• By clicking these icons, you can
• View Contact Information
• Email Primary Contact

54
(No Transcript)
55

• CATS
• The term CATS is used on the CODESP website
  frequently and is also found throughout this
  guide. CATS is an acronym that stands for CODESP
  Automated Test System.
• This section will cover the following topics
• Requesting Test Materials
• Picking up Test Material Packets
• Building your Test
• Reusing Tests
• Reports

56

• Requesting Test Materials
• The CATS Request Form is an order form used to
  order test materials such as multiple choice test
  questions, supplemental application forms, and
  interview questions, etc..
• To make a request, simply follow each step
• Click on CATS Request Form listed on the
  navigation bar.
• Read the CATS Information Request Directions.
• Click on Fill Out Request Form.

57
CATS Request Form Contd

• The CATS Information Request Form screen will
  appear. An example is listed below.
• Enter a job title.
• Select a month and day from the drop down menu to
  indicate when the materials are needed by.
• We require a MINIMUM of 7 Business Days
• Provide CODESP with specific information and
  instructions about the position, so CODESP can
  collect test materials that are suitable for your
  district.
• Click Submit when you are ready to submit your
  request.

58
CATS Request Form Cut and paste your job
description into the Comments Box. Provide
details if your description is vague. For
example Name the specific software required if
your job description only states Computer
skills required.
59

• Picking up Test Material Packets
• Based upon the job description and/or the special
  instructions listed on the CATS Request Form,
  CODESP will create a packet of test questions
  from their question database and send them to
  your account. The packet of testing materials
  will be accessible online.
• When the packet of test materials has been
  completed, CODESP will send an email notification
  to the individual specified as the contact person
  on the CATS Request Form. The email subject line
  will read CATS Materials Request Confirmation.

60
Email Notice You Will Receive

• Your CATS Request with the job title of 'Another
  Sample Request Worker' has been processed and
  your materials packet is ready for your review.
  (ignore numbers identifiers for packets)To
  review your materials, log in to Codesp and click
  on the 'Create a Test' link from the left-hand
  navigation menu

61

• Picking up Test Material Packets Contd
• To pick up your packet
• Log onto your account.
• Click the hyperlink under My Request Status and
  the following will appear.

62
Tracking Your Request

• You can also track the status of your request by
  looking at the My Request Status section found on
  the start page. Your request will be displayed
  as one of the following
• New Request has not been processed by CODESP.
• In progress Request is currently being processed
  by CODESP, but it is not complete.
• Finished Request has been completed. Request
  now called packet. Click on hyperlink. It will
  be stored under Create a Test on the webpage.

63
Training Registration Reminders are listed on
your CATS home page. Also, My Request Status lets
you know when your Request is Ready.
Note The My Request Status section will only
display the status of your requests that have
been made in the last 30 days.
64
To view all packets that have been processed,
click on Create a Test found on the navigation
bar.
65

• Building Your Test
• Now it is time to generate your test! To begin
  the CATS Test Generator, follow these steps
• Click on Create a Test, which can be found on the
  navigation menu.
• Choose the packet you would like to work on by
  clicking the position title.

66
(No Transcript)
67

• Building your Test Contd
• Provide a title for the test.
• Allocate how much time the candidate will be
  given to take the test.
• Click Next when you are ready to move onto the
  next screen.

68
Custodian
69

• Building your Test Contd
• Review all questions in packet. Note The
  correct answers are indicated with a green check
  mark.
• To add question to your test, check the box under
  Add to Test.
• Optional Sort questions in the order you wish
  them to appear on test (change test number).
• Click Create a Test to complete this process.
• If you are not satisfied with the materials in
  the packet, call us first and we will advise you
  to click Send Back or to submit a new Request.

70
All Questions which begin with an F require you
to click on the Corresponding Figure above the
item (Figure lose their item numbers during the
print process) See CATS Tutorial for more
details.
71

• Building your Test Contd
• If you Send Back the items explain to CODESP why
  these materials did not meet your needs and what
  subject area you need additional questions from.
• CODESP will then rebuild the packet based on your
  comments. If it is a Totally Wrong Packet-Call us
  we will advise you if you should submit a new
  Request.
• Send Backs can be avoided if you provide us with
  accurate information about the job on the Request
  Form.

72
Reusing Tests After you have generated a test,
the test will be stored in the Test Library.
Each test can be reused by clicking on the
printer icon.
The Red X will delete the test. Only delete
tests that were created as practice. Tests that
were administered should remain in your Library,
a long with a hard copy in a LOCKED cabinet. We
suggest regenerating your test each time you
administer it to receive the latest updates.
73
Printing a Test

• By clicking on the Printer Icon you can print the
  test from your desk top.
• Follow the instructions as you go through the
  print process.
• Enter the time you want for your test.
• Make sure you Click All Pages.
• Make sure your title is correct.
• Check the test before administering
• Are all pages printed
• Are the figures easy to read
• Did all of the figures appear above the
  corresponding items.

74
Interviews, Supplementals, Performance Tests and
Writing Exercises

• Samples are available in Word format in the
  Resource Library.
• Scroll down to the type of test.
• Click on the subject area.
• Click on the title which most closely matches the
  title of the position you are requesting sample
  test materials for.
• Open a separate Word document and copy and paste
  the questions from the samples provided to your
  blank Word document.

75
Resource LibraryClick on the type of documents
or test materials needed to expand the selection.
76
CODESP SAMPLE INTERVIEW QUESTIONSACOUNTING
TECHNICIAN/AUDITORSAMPLE INTERVIEW
QUESTIONS 1. Describe your experience auditing
business records to determine unemployment
insurance premiums, liabilities, and district's
compliance with federal and state laws. How
extensive were your audit reports?2. Describe
your experience reconciling District's internal
records with the County PBAS System. (Add
whatever system is required for this
position).3. Describe how you have utilized your
knowledge of the State of California Accounting
Manual for Local Education Agencies in conducting
school district audits. 4. Describe your
experience auditing and reconciling state and
county revenue limit reports.
77
After copying and pasting into your own Word
Document you may edit the questions to fit your
districts needs such as the following sample
using the first 2 questions from the previous
slideACCOUNTING TECHNICIAN
1. Describe your experience auditing business
records to determine unemployment insurance
premiums, liabilities, and district's compliance
with federal and state laws. How extensive were
your audit reports?2. Describe your experience
reconciling district's internal records with the
County SAS System.
78
Pre-Assembled Tests

• Pre-Assembled tests for positions such as
  Custodian (entry and experienced), Food Service
  Worker and Food Service Cashier, Clerk,
  Instructional Aide, and Secretary are be
  available for viewing in the Resource Library.
• You can use them as is or order a custom test by
  completing a CATS Request Form.

79

• District Administrator Account
• District Administrator accounts are unique. A
  person who has District Administrator access has
  additional features on their navigation menu,
  which gives them access to
• Add/Edit Profiles
• Edit District Profile
• In the illustration that follows, it is pointed
  out that all the links below the dashed line are
  features only accessible by people who have a
  District Administrator account. Links above the
  dashed line are available to all users.

80
This section will cover the following topics Add
New User Profile Edit User Profile Edit District
Profile
81

• Add New User Profile
• Upon registering for services, CODESP creates one
  District Administrator account. Once this
  account has been created, it is the District
  Administrators responsibility to create
  additional accounts for other district employees.
• Add User Profile
• Click on Add/Edit Profiles
• Click on the Add New User button located in the
  bottom right-hand corner.
• The Add New User screen will appear.

82

• Add User Profile Contd
• On the Add New User screen, assign user a login
  name and password.
• Enter users contact information.
• Define users Security Role by selecting
  Application User or District Administrator (does
  not have to be an actual Administrator.
• Click Update Profile to save.

83
(No Transcript)
84

• Edit User Profile
• The District Administrator is charged with
  maintaining user profiles. As a reminder, listed
  below are common occurrences that should raise a
  red-flag to go in and amend the users profile
• Person changes positions
• Person retires
• Contact information changes
• Person changes their name
• Person will be taking a test for another job in
  district
• Edit Profile
• Click on Add/Edit Profile located on the
  navigation bar.

85
(No Transcript)
86

• Edit Profile Continued
• Make appropriate changes.
• Click Update Profile.

87
Edit District Profile The District Administrator
is also responsible for keeping their district
contact information current. This information is
found by clicking Edit District Profile. After
you have made your changes, click Update Profile.
88
CATS Training

• CATS Tutorials with more detailed instruction are
  located under Resources / Tutorials
• Give us a call if you cant find the answer
  on-line
• 714-374-8644
• We are continuously researching and/or adding new
  features

89
QUESTIONS??

• Email tests_at_codesp.com
• www.codesp.com