Assessment Integrity: The Dark Side of Distributed Learning

1
Assessment Integrity The Dark Side of
Distributed Learning

• Kirk Wm McCullough, Ed. S. Marsha M. Jurewicz,
  Ed. D.

2
Problem Statement

• Become aware of some elements of cheating, and
  a method of approach, so distributed learning
  resources can be more appropriately applied.

3
Points of Paper

• People cheat in traditional training
• Importance varies with context
• People carry that behavior to online training
• Prevention methods have costs
• How do you determine an appropriate response?

4
Risk Assessment Model

• Systems Characterization
• Threat Identification
• Vulnerability Identification
• Control Analysis
• Likelihood Determination
• Risk Determination
• Control Recommendation

5
Systems Characterization

• Scope of the effort your command, your school
• Types of assessments
• Pre-test
• Quiz
• Periodic
• End-of-training
• Certification
• Delivery methods
• Paper
• SCO
• Practical exam
• CBT

6
Threat Identification

• Methods of cheating
• Collaborative cheaters
• Unauthorized access
• Physical
• Technical
• Assessment design
• Plagiarism
• Identify opportunities

7
Vulnerability Identification

• Lack of supervision
• Preservation of answers
• Uniformity of questions and answers
• Accessible answers
• Institutional policies

8
Risk Assessment Model

• Systems Characterization
• Threat Identification
• Vulnerability Identification
• Control Analysis
• Likelihood Determination
• Risk Determination
• Control Recommendation

9
Control Analysis

• Binary coding
• Access controls
• Randomized questions
• Technical threat controls
• Off line exams

10
Likelihood Determination

• How do you determine a reasonable probability?
• Assume 100
• Assume equivalent probability
• Assume lesser probability

11
Impact Analysis

• Effects on learner
• Artificial self-inflation
• Loss of self-esteem
• Decrease in collaboration
• Morality/ethics practice

12
Impact Analysis

• Effects on organization
• Mission impact
• Less useful personnel
• Less course utility
• Lower morale/morality/ethical behavior
• Less reliable certification

13
Risk Determination

• General Assessment Risk matrix
• Likelihood x Impact Risk
• Quantification may be a problem
• Acceptable risk

14
General Assessment Risk
15
Examples of Control Recommendation Factors

• Is collaboration desirable?
• Is independent work required?
• How critical is exam score to mission?

16
Control Recommendations

• Identify control options
• Weigh the control costs
• Identify decision makers
• Identify control methods
• Identify control impact on personnel and facility

17
Future Studies

• Does rate of cheating differ Civilian/DoD?
• Self reported
• Designed study
• Measures of effectiveness
• Reminders to do own work
• Technical safeguards

18

• Questions ??

19
Experience. Results.

• Kirk McCullough [phone] kmccullo_at_csc.com
• Marsha Jurewicz [phone] mjurewicz_at_csc.com