Development Of An Army Assessment Test: Experiences and Lessons Learned

1
Development Of An Army Assessment Test
Experiences and Lessons Learned

• Roy Campbell
• Karen Moriarty
• Tonia Heffner

2
Demonstration Competency Assessment Program (DCAP)

• PURPOSE Conduct feasibility development of a
  prototype, computer-based, web-administered test
  of Army-wide (non-job specific) competencies for
  promotion to the junior NCO (E5) ranks.

3
Test Development Steps
4
SITUATION Job Analysis

• Resources precluded detailed job analysis.
• Relied on senior NCO leadership for designation
  of broad content.
• Conducted workshops to define blueprint.
• Content areas
• Basic Soldiering (Common Tasks weapons, first
  aid, navigation, NBC)
• NCO/Army History, Customs, Army Values
• Leadership
• Training
• Observation Sound job analysis remains the most
  important single part of test development

5
SITUATION Test Blueprint
6
SITUATION Test Blueprint

• Observation
• The Army-wide test blueprint needs to be a
  flexible document, defining broad areas of
  performance complete revision on an annual
  basis is neither feasible nor desirable. Work
  is needed in category definition and description,
  particularly in doctrinally weak areas.

7
SITUATION Item Tracking

• Within the Basic Soldiering tasks, some tasks
  (particularly weapons related) are unit or
  assignment specific.
• With a computerized test, tracking Soldiers to
  specific content areas based on an experience or
  selection question is relatively simple.

8
SITUATION Item Tracking

• Observation
• Tracking is not the preferred approach
• Equivalency and comparability
• Self-select to easier tracks
• Criteria for tracking decisions
• Increased development and maintenance
• Avoided tracking in DCAP
• Collecting experience data

9
SITUATION Non-traditional Items

• Computer based test construction and
  administration allows options beyond four-choice,
  one-correct traditional approach
• Matching
• Drag and drop
• Ranking
• Select all that apply
• Item counts lose meaning
• DCAP Draft 282 items with 364 scoring points

10
SITUATION Non-traditional Items

• Observation
• Non-traditional items are controversial
• Reviewers object to format changes, however
  Soldiers like the variety.
• New scoring schemes and protocols are required.
• Blueprint coverage and reliability becomes an
  issue.
• Item dependence needs to be factored.

11
SITUATION Notification and Preparation

• DCAP goal was 2 month notification of Soldiers
  via Army Knowledge Online (AKO) web addresses.
• Direct Soldiers to a web site with test
  information, sample items, electronic access to
  references.

12
SITUATION Notification and Preparation

• Observation
• 80 of pilot test Soldiers notified lt 1 week.
• About 1 of pilot test Soldiers did any
  preparation.
• However, they did endorse the preparation
  materials.
• Many Soldiers do not use, or even know, their AKO
  addresses.

13
SITUATION Digital Training Facilities (DTF)

• Army has 200 DTF locations
• Each suite has 16 computers some locations have
  multiple suites.
• Access to Internet and Army Distance Learning
  System (DLS) network.
• Soldiers must be registered to log on and use.
• Used by AC and USAR ARNG has comparable but
  separate system.

14
SITUATION Digital Training Facilities (DTF)

• Observation
• DTFs have all the necessary components to conduct
  computer based, web based testing.
• DTF mission is distance learning (DL)
• Once underutilized, demands on DTF are rapidly
  increasing.
• Operational large scale testing would likely tax
  DTF resources
• Proctoring remains an issue.

15
SITUATION Technical Problems

• More technical issues than anticipated
• Log-in problems.
• Computer specific problems
• System problems software
• Variability of problems by site.
• Observation Soldiers must have trust in the test
  system. Resolving technical issues is an absolute
  requirement.

16
SITUATION Soldier Reactions
17
Summary and Conclusions

• Overall, significant achievements have been made
  in design and in identification of operational
  issues and requirements.
• Some problem areas must continue to be addressed
  before critical implementation decisions can be
  made.
• In testing of this scale and impact, getting it
  right is more important than getting it done.