Research on Universal Design of Assessments

1
Research on UniversalDesign of Assessments

• Making Assessments Accessible
• and Valid for All Students

2
Universal Design of AssessmentThe Whole Elephant

• David Malouf
• Office of Special Education Programs
• U.S. Department of Education

3
Universal Design of Assessments
Designing assessments to be inherently accessible
and valid for the widest possible range of
students.
4
Series of Federal Laws

• Improving Americas Schools Act of 1994
• 1997 reauthorization of the Individuals with
  Disabilities Education Act
• No Child Left Behind Act of 2001
• Upcoming reauthorization of the Individuals with
  Disabilities Education Act

5
Improving Americas Schools Act of 1994

• Under Title I, all students were to be included
  in assessments, public reporting, and
  accountability.

6
Individuals with DisabilitiesEducation Act (1997
reauth.)

• Added requirements for students with disabilities
  to be included in State and district-wide
  assessments.
• Introduced alternate assessments
• Required public reporting of aggregated and
  disaggregated data

7
No Child Left Behind Act of 2001

• Increased assessment demands
• Increased requirements for including students
  with disabilities and students with limited
  English proficiency
• Accountability for subgroups

8
From the Regulations for theNo Child Left Behind
Act
200.2 State responsibilities for
assessment (b) The assessment system required
under this section must meet the following
requirements ....... (2) Be designed to be
valid and accessible for use by the widest
possible range of students, including students
with disabilities and students with limited
English proficiency.
9
Presidents Commission on Excellence in Special
Education

• Created in 2001 to recommend reforms in Americas
  special education system
• 13 Public hearings and meetings, plus written
  input
• Reported in July 2002

10
Presidents Commission on Excellence in Special
Education

• RECOMMENDATIONINCORPORATE UNIVERSAL DESIGN IN
  ACCOUNTABILITY TOOLS Ensure all tools used to
  assess students for accountability and the
  assessment of progress are designed to include
  any accommodations and modifications for students
  with disabilities.

11
Upcoming reauthorization of the Individuals with
Disabilities Education Act (IDEA)
?
12
Senate Bill 1248

• Universal design. The State educational agency
  (or, in the case of a districtwide assessment,
  the local educational agency) shall, to the
  extent possible, use universal design principles
  in developing and administering any assessments
  under this paragraph.
• 612(a)(16)(A)(ii)(E)

13
The pressure is on...
Will we do quick retrofits of tests and called
them universally designed?
Will we simply port tests onto computers and call
them universally designed?
14
Constructs designed for accessibility
Inclusive Population
Accessible items
Amenable to accommodations
Simple and clear instructions and procedures
Maximum readability
Maximum legibility
15
Three Research Projects on Universal Design of
Assessment

• Development Techniques for Universally Designed
  Assessments
• Sandra Thompson, University of Minnesota
• Access to Assessment via Technology
• Jacqueline Kearns, University of Kentucky
• Bob Dolan, CAST
• Project MAP (Making Accommodations Personalized
• Gerald Tindal, University of Oregon

16
Development Techniques for Universally Designed
Assessments
National Center on Educational
Outcomes University of Minnesota Sandy Thompson
http//education.umn.edu/nceo
17
3 Studies

• Use think aloud to examine student end user -
  perspectives
• Analyze items for differences possibly due to
  design features
• Design training for item/test reviewers

18
Think aloud

• Recently interviewed 90 students using think
  aloud protocol
• 4th and 8th grade
• Used multiple choice and constructed response
  items from state math test

19
Logistics

• 10 researchers
• 5 days
• Worked in pairs
• All sessions videotaped
• Primary accommodations included oral
  administration and sign language interpretation

20
Student Characteristics
Grade 4 Grade 8
Learning Disability 10 10
Deaf/Hard of Hearing 10 10
Mild Cognitive Impairment 5 5
English Language Learner 10 10
No Disability 10 10
21
Overall Observations

• Students who were confident of content did not
  have problems with design
• Students who had no idea how to solve the problem
  did not have problems with design
• Students in the middle not sure of content,
  some reading difficulty, design made a difference

22
Examples of Student Perceptions

• Many students didnt see one of the cities on a
  map
• The name of one of the cities was Independence
  - uncommon meaning
• Box between top and bottom of item some
  students did not read entire item
• Sign for parallel gave away the answer
• Some students read fraction 3 5/8 as 35 divided
  by 8
• Students unfamiliar with settings Glee club
  does number, fitness club

23
Other Observations

• Some students got a lot of help from sign
  language interpreters and teachers who wanted to
  make sure they understood the problem what
  happens on test day?
• Some student forms reported need for oral
  administration when they could clearly read the
  items independently are some accommodations
  inappropriate?
• For English language learners - is oral
  administration in English an appropriate
  accommodation on a Math test?
• Should sign language interpreters have a script
  to follow so they dont give away some answers?

24
Considerations for Item Review

• Overall appearance is clean and organized
• Clear format for text
• Clear format for pictures and graphics (when
  essential to item)
• Concise and readable text
• Format allows for changes without changing
  meaning or difficulty
• Meets criteria for measuring what it is intended
  to measure

25
Considerations for Test Review

• Meets general criteria for measuring what it is
  intended to measure
• Overall appearance is clean and organized
• Instructions are necessary, clear, and
  understandable
• Scoring criteria are appropriate
• Others?

26
Alignment and Usability Need to be Considered
Together
27
Future Plans

• Interview students with vision impairments
• Partner with American Printing House for the
  Blind
• Explore differential item analyses
• Partner with researchers from CTB/McGraw-Hill
• Produce short awareness video
• Partner with Institute on Community Integration
  at University of Minnesota
• Design and pilot training for item reviewers and
  item developers
• Partner with Missouri Department of Education

28
Universal Design of Assessment Applications of
Technology
CCSSO National Conference onLarge-Scale
Assessment June 23, 2003
29
Universal Design for Learning (1)

• Universal Design for Learning (UDL) extends the
  concept of universal design from a physical space
  to a cognitive space
• Based upon psychological and neuroscientific
  theories of learning
• Relies on scaffolds, supports, accommodations
  which support students challenges and thus
  provide access to learning
• Applies to design and development of goals,
  methods, assessments, and materials

30
Universal Design for Learning (2)

• To support individual differences, students must
  be provided with multiple means of interacting
  with curriculum
• Multiple means of representation
• Multiple means of interaction and expression
• Multiple means of engagement
• Digital technology is neither necessary nor
  sufficient for UDL, but it is an enabling factor

31
Kentucky

• Universal Design for Learning Expert Group
• Instructional Technology for Student Success
  (ITSS) initiative
• Digital Text Network
• E-Text Schools program
• Senate Bill 243
• Requires any textbook offered for adoption in
  Kentucky schools be made available by the
  publisher in an accessible digital format

32
KY CATS Online Assessment

• Web-based, individualized assessment for
  qualifying students with disabilities
• Students with IEP or 504 Plan that specifies need
  for "reader" as an instructional and assessment
  accommodation
• Students who require and routinely use
  text-reader or screen-reader technologies to
  access printed material in classroom instruction
  and assessment
• Students who have accessed and used the CATS
  Online Practice Area.
• Based upon success of pilot studies, 16
  districts, 31 Schools, 204 students
  participated in live CATS Online this spring

33
CATS Online
34
Maine

• Maine Learning Results emphasize inclusion of
  diverse students by separating goals and methods
• Maine Comprehensive Assessment System uses
  variety of state and local components, allowing
  flexibility and multiple measures of learning to
  accommodate learner diversity
• State assessment (MEA) tightly aligned with
  Learning Results
• Actively pursuing UDL as a means for increasing
  access to general curriculum

35
Overarching Research Question

• How can digital technologies be used to improve
  accessibility of large-scale assessments in
  accordance with the principles of Universal
  Design for Learning?

36
Research Question 1

• What are the practical, procedural, and political
  design and implementation features of accessible
  computer-based assessments?

37
Research Question 2

• What technology pre-requisite skills do students
  need to use computer-based assessments?

38
Research Question 3

• What is the impact of a computer-based
  universally designed assessment on the scores of
  students with disabilities?

39
Research Question 4

• To what extent does accessible curriculum design
  impact student assessment results?

40
Research Question 5

• Can the universally designed assessment design
  features be replicated with new content in a
  different state context?

41
Research Methodology

• Qualitative Studies
• Interviews, observations, focus groups
• Quantitative Studies
• Survey, score analysis
• Critical Theory
• Constituent reflection

42
Preliminary Findings To Date

• Interviewed approximately 40 students from 7
  schools who took CATS Online
• Most students used the text-to-speech feature
• Students liked the independence that the text
  reader provided, which allowed them to re-read
  questions multiple times

43
Features that Promote Accessibility

• Multiple means of representation
• Text-to-speech
• Font features (e.g. size)
• Screen layout options
• Textual description of images
• Multiple means of expression and interaction
• Word processing
• Flexible navigation

44
Features that Impede Accessibility

• Word processing requires a set of skills that
  some students may not have
• Text-to-speech voice quality and accuracy
  issues
• Layout issues poetry, tables, etc.

45
Preliminary Item AnalysisPositive Exemplars
46
MEA Grade 11 Science and Technology
16. This diagram shows that the beams of light
from two flashlights can pass through each other
and then continue on unaffected. This observation
illustrates which property or properties of
light? A. only particle ? B. only wave C.
both particle and wave D. neither particle nor
wave
Learning Results H-1, Energy. Students will
understand concepts of energy. Students will be
able to analyze the evidence that leads
scientists to conclude that light behaves
somewhat like a wave and somewhat like a particle.

• Consider
• Explicit reference to image
• Redundancy of information between image and text

47
Scales of Justice Use the illustration below to
answer question 12.
CATS Grade 8 Social Studies
12. Lady Justice is a symbol of the United
States justice system. As shown in the
illustration above, she wears a blindfold and
carries a balance scale. Explain how this symbol
represents two characteristics of the U.S.
justice system.

• Consider
• Explicit reference to image
• Redundancy of information between image and text
• Multiple opportunities for response (writing vs.
  keyboarding)

48
Preliminary Item AnalysisNegative Exemplars
49
MEA Grade 11 Math
2. Which of the following is the best estimate of
the sum of
.v26 v78? A. 10 ? B.
14 C. 52 D. 104
Learning Results B-1, Computation. Students will
understand and demonstrate computation skills.
Students will be able to use various techniques
to approximate solutions, determine the
reasonableness of answers, and justify the
results.

• Consider
• Ability of text-to-speech tools to recognize and
  represent the square-root symbol
• Potential invalidation of item when square-root
  symbol is decoded by human or digital reader

50
Alexs Garden
CATS Grade 8 Math
30. Alex is watering part of his garden with a
sprinkler that covers a circular area shown in
the diagram below. a. What is the total area of
Alexs garden? Show your work. b. What is the
area of the part of the garden that is being
watered by the sprinkler? Express your answer to
the nearest square foot. Show your work. c. What
percent of his garden is being watered by the
sprinkler? Express your answer to the nearest
percent. Show your work. BE SURE TO LABEL YOUR
RESPONSES (a), (b), and (c).

• Consider
• Transferring of stimulus to paper for response
• Use of text-to-speech with image labels
• Representation of image to blind students

51
Concerns Conclusions

• Applying universal design to instruction and
  assessment which comes first? (Chicken and egg
  problem)
• Practical considerations of use of technology
  e.g. security, scalability, cost
• Need for seamless integration of instructional
  and assessment technologies to ensure valid,
  accurate, and fair testing