The Learning Side of Testing

1
The Learning Side of Testing

• Milwaukee Area Technical College
• Dental Hygiene Program
• February 25, 2005

2
Test to Learn
What does this mean? What does it tell us about
how testing fits into performance-based learning?

• Learn to Test

3
Situational Analysis
Plus Minus Interesting
National Board Results
Testing Practices
Other
4
Situational Analysis
What do we know about National Board Results?
MATC performance in comparison to all others.
5
Situational Analysis
What do we know about National Board Results?
MATC failure rate.
6
Situational Analysis
What do we know about National Board Results?
7
Situational Analysis
What do we know about testing practices?
8
Situational Analysis
What else might impact the issues?
9
Testing as Learning
Testing promotes learning. Testing requires
learning.
Learning requires practice!
10
Practice and Learning
Sensing Memory
Three types of memory

• selector
• large capacity
• short duration

11
Practice and Learning
Working Memory

• processor
• powerful
• fragile
• short duration
• small capacity

Three types of memory
12
Practice and Learning
Long-Term Memory
Three types of memory

• information storage
• large capacity
• requires retrieval

13
Five mental processes control information flow
attention (sensing) encoding (working/long-term)
process/practice (working) retrieval
(long-term/working)
14
Practice and Learning
Learning is enduring and meaningful only when the
Three types of memory
sensing memory attends to the stimulus
working memory processes it and sends it to
long-term memory
long-term memory stores it
15
Practice and Learning
Aha! Its an occlusal.
Learners apply what theyve learned only when
they can retrieve it from long-term memory .
Three types of memory
16
Practice and Learning
Research has found that providing learners with
retrieval practice is much more powerful than
giving them the same amount of time to restudy
the learning material. Typical improvements
range from 30 to 100.
Will Thalheimer Simulation-Like Questions How
and Why to Write Them www.work-learning.com
17
Practice and Learning
. . . hmm, the answer to this question isnt what
I thought it was. What do I need to do about it?
Metacognition
Learners need feedback in order to take
responsibility for monitoring and adjusting their
own learning.
18
Practice and Learning
Feedback is essential when learners answer
questions because it helps them correct their
misconceptions. Feedback typically improves
learning outcomes by 15 to 50.
Will Thalheimer Simulation-Like Questions How
and Why to Write Them www.work-learning.com
19
Practice and Learning
Too much information too fast yields
Fried Brain Syndrome
20
Practice and Learning
Fried brain syndrome leads to
Cognitive Overload
Cognitive Overload
Cognitive Overload
Cognitive overload slows down or even stops
learning!
21
Practice and Learning
To avoid cognitive overload
Three types of memory

• break learning into manageable chunks
• clear the working memory with meaningful practice

22
Performance vs. Practice

• Intended to accomplish a task
• Real thing/big bites
• On your own
• High stakes
• Feedback after the fact
• Limited opportunities to correct
• School of hard knocks

• Intended to promote learning
• Protected/chunked
• Coached
• Low stakes
• Concurrent or immediate feedback
• Repetition
• Built-in opportunities to correct

23
Testing as Learning
Tool Description Measure vs. Learn Stakes
Educational Assessment Process of obtaining evidence about the learners knowledge, skills, and attitudes.
Exam Measure/document knowledge or skills for the purpose of credentialing (graded). Measure High
Test Measure knowledge or skills for the purpose of informing the learner and teacher about the learners current level of knowledge or skill. (likely to be graded) Measure- Promote Learning Medium
Quiz Measure a learners knowledge or skills for the purpose of providing feedback and identifying gaps. Promote Learning Low
CAT Provide immediate feedback to teachers and learners to guide adjusts in teaching and learning. Promote Learning Very Low
Slide from derived http//questionmark.com
24
Testing as Learning
What does this mean?
Testing provides practice. Testing requires
practice.
25
Testing as Learning
Strategies for Using Testing as Learning

• Require learners to test drive their knowledge
  by completing low stakes quizzes at all stages of
  learning. Could be self-check.
• Formulate classroom quizzes and tests using the
  same test formats used in the National Boards.
• Present learner with questions that require
  higher level thinking to make decisions,
  determine solutions, or draw conclusions.

26
Testing as Learning
Strategies for Using Testing as Learning

• Design questions so that learners apply knowledge
  in the context of Dental Hygiene practice.
• Provide corrective feedback for incorrect
  answers.
• Build in branching to present additional
  questions after missed questions.
• Teach learners early and often how to succeed on
  tests and quizzes.

27
What do you know about writing multiple choice
test items?
Question Writing Process

1. What should you do?
2. What makes a good question?
3. What should you avoid?

28
Question Writing Process

• Start with your competencies and learning
  objectives
• Write question stems
• Write correct answers
• Write distractors
• Edit

29
Start with target competencies and learning
objectives to

• ensure validity
• ascertain significance
• check coverage
• determine scope of test

30
Write the Stem

• Derive the question from a competency or learning
  objective.
• Identify target decision, conclusion, or
  solution.
• Present a single, definitive problem with a
  definitive answer.
• Make stem grammatically consistent and parallel
  in form to options.

31
Write the Stem
What to avoid

• Negatively phrased questions (used in National
  Boards)
• Absolutes such as always and never
• Unnecessary information
• Obvious clues about the answer
• Bias (gender, cultural, ethnic, regional)
• Humor and trick questions

32
Avoid Grammatical Clues

• Option A is the only option that makes
  grammatical sense.
• The best way to increase the reliability of a
  test is to
• Increase the test length. Correct
• Removing poor quality items.
• Test should be readable for all test takers.

33
Avoid Association Clues

• Option B includes the word seven.
• Who are the Magnificent Seven?
• A pro basketball team Correct
• A group of seven fictional, heroic, Western
  figures Correct
• A protest group made famous during the 1960s
• A vegetarian rock group

34
Write the Options

• Write three to five options.
• Make all options grammatically consistent and
  parallel in form to stem.
• Write all options so they are comparable in
  content, length.
• Ensure that options are not repetitious.
• Order them logically or numerically.
• Avoid All of the above and None of the above.

35
Order Options Numerically
What is the cost of an item that normally sells
for 9.99 USD that is discounted 25?

• Right
• 2.50
• 5.00
• 6.66
• 7.50

• Wrong
• 5.00
• 7.50
• 2.50
• 6.66

36
Order Options Logically
What are the three important concerns in
achievement testing? Vvalidity Rreliability
Eefficiency Oobjectivity

• Right
• V, R, O
• V, R, E
• V, E, O
• R, E, O

• Wrong
• V, R, E
• O, V, R
• R, O, E
• E, O, V

37
Write the Options - Distractors

• Write two to four distractors (giving 3-5
  options).
• Make all distractors plausible.
• Make distractors similar in form or otherwise
  harmonious with the answer.

38
Write the Options - Correct Answer

• Be sure that correct answer is the only correct
  one or clearly the best one.
• Vary the correct answer position in the list of
  options.

39
Raising the Level of Thinking

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

Critical Thinking
40
Raising the Level of Thinking

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

define label list recall recognize
41
Raising the Level of Thinking

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

explain paraphrase describe summarize
42
Raising the Level of Thinking

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

apply decide determine use
43
Raising the Level of Thinking

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

classify compare describe structure determine
cause/effect detect patterns diagnose
44
Raising the Level of Thinking
construct create design develop formulate write

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

45
Raising the Level of Thinking
appraise assess conclude evaluate prescribe

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

46
Raising the Level of Thinking
How do you help learners practice higher level
thinking skills?
Continually present questions that require them
to

• make a decision
• draw a conclusion
• solve a problem

47
Tools

• Higher Level Question Worksheet
• Case-Based Question Planner

48
PerceptionA Web-based Tool for QA Assessment
Report on the results. Reports include coaching
reports, item analysis, gap analysis, and others.

• Create questions and group into quizzes, tests,
  exams and surveys using a web browser or Windows
  PC

Deliver assessment using a standard, or secure,
browser, or using a Windows PC
Specify security parameters, schedule
assessments and potentially link to other systems
49
The Dental Hygiene National Boards
0
High Stakes Exam
Slide from http//questionmark.com
50
The Dental Hygiene National Boards
Context and Purpose

• Intended to assess ability to apply knowledge in
  a problem-solving context
• Key to qualifying for licensure
• Developed by dental hygiene and science educators
• Subject to on-going validity and reliability
  checking
• Administered in highly secure environment with
  significant consequences for cheating and breach
  of security
• Administered according a consistent set of strict
  rules
• Administered to a large, diverse population
  across a broad geographic area.

51
The Dental Hygiene National Boards
Scope

• Discipline-Based (200 questions)
• Scientific Basis for Dental Hygiene Practice
• Provision of Clinical Hygiene Services
• Community Health Services
• Behavioral Sciences (infused into 5)
• Ethics and Risk Management (infused into 5)
• Cased-based items (150
• Assessing patient characteristics
• Obtaining and interpreting radiographs
• Planning and managing dental hygiene care
• Performing periodontal procedures
• Using preventative agents
• Providing supportive treatment service

52
The Dental Hygiene National Boards
Scope

• Discipline-Based (200 questions)
• Scientific Basis for Dental Hygiene Practice
• Provision of Clinical Hygiene Services
• Community Health Services
• Behavioral Sciences (infused into 5)
• Ethics and Risk Management (infused into 5)
• Cased-based items (150
• Assessing patient characteristics
• Obtaining and interpreting radiographs
• Planning and managing dental hygiene care
• Performing periodontal procedures
• Using preventative agents
• Providing supportive treatment service

53
The Dental Hygiene National Boards
Format Multiple Choice
Seven Question Types

• Completion
• Question
• Negative
• Paired True-False
• Cause-and-Effect
• Testlet
• Case-Based

54
The Dental Hygiene National Boards
Format Completion Typical Level
Knowledge
Example
Radiographic intensifying screens are used for

1. magnifying lamps.
2. increasing detail.
3. reducing the exposure time.
4. speeding processing time.
5. decreasing the processing time.

55
The Dental Hygiene National Boards
Format Question Typical Level
Knowledge, Understanding, Application (all)
Example
Root hypersensitivity diminishes as the tooth
forms which of the following?

1. Mantle dentin
2. Secondary dentin
3. Cellular cementum
4. Acellular cementum

56
The Dental Hygiene National Boards
Format Negative Typical Level
Knowledge, Understanding, (all)
Example
Each of the following characterizes an ideal
chemical disinfectant EXCEPT one. Which is the
EXCEPTION?

1. Heat-stable
2. Long shelf-life
3. Water solubility
4. Activity against vegetative microbes
5. Activity against microbial spores

57
The Dental Hygiene National Boards
Format Paired True-False Typical Level
Understanding, Analysis, Evaluation
Example
Protection from excessive exposure to radiation
is aided by the use of aluminum filters and a
lead diaphragm. The filters reduce the amount of
soft radiation.

1. Both statements are TRUE.
2. Both statements are FALSE.
3. The first statement is TRUE the second is FALSE.
4. The first statement is FALSE the second is TRUE.

58
The Dental Hygiene National Boards
Format Cause-and-Effect Typical Level
Analysis
Example
Adolescent growth spurts more in the maxilla than
in the mandible BECAUSE, at puberty, the lymphoid
tissue present in the nasopharynx decreases.

1. Both the statement and the reason are correct.
2. The statement is correct, but the reason is NOT.
3. The statement is NOT correct, but the reason is
   correct.
4. NEITHER the statement NOR the reason is correct.

59
The Dental Hygiene National Boards
Format Testlet (Community Health) Typical Level

Application, Analysis, Evaluation
Description
Presents a community health scenario followed by
five or more multiple-choice items that address
its content. Questions are likely to be in the
question format, but could be in any format. For
an example see 2005 National Board Dental Hygiene
Examination Candidate Guide, p. 40.
60
The Dental Hygiene National Boards
Format Case-Based (Patient Care) Typical Level

Application, Analysis, Evaluation, Synthesis
Description (12-15 on exam)
Present patient histories, dental charts,
radiographs (sometimes intra- and extra-oral
photographs). Test items, typically in the
question format, address functions that a dental
hygienist is expected to be able to
perform. Exams will include at least one case
regarding the following types of patients
geriatric, adult periodontal, pediatric, special
needs, medically compromised. For an example see
2005 National Board Dental Hygiene Examination
Candidate Guide, pp. 41-45.
61
Enhancing Your Tests
Critique existing tests.

• Validity
• Test items
• Feedback
• Use of results

62
Enhancing Your Tests
Critique existing tests.
Strengths Weaknesses Interesting

63
Enhancing Your Tests
Outline test improvement strategies.
New and Improved!
64
Enhancing Your Tests

• Create an assessment plan that includes both
  performance assessment tasks and QA tests.
• Use competencies and learning objectives to plan
  QA tests ensure validity, ascertain
  significance, check coverage and determine scope.

65
Enhancing Your Tests

• Review and revise existing tests and exams using
  the Checklist for Q A Quizzes and Tests.
• Link all test questions to a stated learning
  outcome (ex. competency, learning objective).
• Increase the number of higher thinking level
  questions.
• Incorporate feedback that corrects misconceptions
  and refers learners who answer incorrectly to
  remedial or further learning for each question.
• Incorporate branching to direct learners to
  remedial, similar, or higher level questions, as
  needed. (Online delivery)
• Increase the variety of questioning formats used
  in all quizzes and tests.

66
Enhancing Your Tests

• Consider using a tool like Questionmark to
• support online delivery
• increased security
• authoring of varied question formats
• immediate feedback
• branching for additional testing
• referral to learning materials
• ease of scoring
• analysis of results, and competency/learning
  objective driven test development

67
Enhancing Their Learning
Provide effective practice throughout at all
phases and levels of the learning process.
Elaborative Practice/Rehearsal
Maintenance Practice/Rehearsal

• connects with what learners already know
• applies to solution of problems
• creates something new
• efficiently encodes to long-term memory

• keeps information in working memory
• clutters working memory
• inefficient

Effective practice is elaborative.
68
Enhancing Their Learning
Provide effective practice throughout at all
phases and levels of the learning process.
Elaborative Practice/Rehearsal
Maintenance Practice/Rehearsal

• memorization
• repetition
• recitation
• drill practice

• classification
• analysis
• skill practice
• critique
• problem-solving
• product creation

Effective practice is elaborative.
69
Enhancing Their Learning

• Make them work for it by using learner-centered,
  active learning activities at the comprehension
  phase.
• Guide learners to process information using
  thinking maps and application exercises
• Engage learners in collaborative information
  gathering and processes activities

70
Enhancing Their Learning

• Present Thinking Maps or other visual learning
  tools have learners create visual learning
  tools.
• Engage learners in the process of developing
  higher level thinking questions and case studies.