Universally Designed English Proficiency and Content Area Assessments

1
Universally Designed English Proficiency and
Content Area Assessments
Improving the Achievement of English Language
Learners through Authentic Proficiency
Assessments May, 2004
2
Training Goals

• Determine why we need universally designed
  assessments and for whom
• Identify and give examples of essential elements
  of universally designed assessments
• Identify and apply considerations for item review
  to sample test items

3
A Quick Definition

• Universally designed assessments are built from
  the beginning and continually refined to be
  accessible and valid for the greatest number of
  students, including English language learners!

4
But, what does that really mean?

• Do we want to change the standard of
  performance? NO
• Can we forget about accommodations if we do
  this? NO
• Is this all figured out for now and
  forever? NO
• Is this something that will benefit only English
  language learners? NO

5
Think about universal design in architecture and
tool design

• Curb cuts and ramps
• Signage with universal symbols
• Door handles rather than knobs
• Special pen shapes that are easier to hold

6
The goal is to provide optimal standard
assessment conditions for todays diverse
population of students
7
Remember this? OFFICIAL BALLOT, PALM BEACH
COUNTY, FLORIDA

8
Analysis of Florida VotesPrecincts with
primarily Hispanic, African American, or
elderly voters had more than 3 times as many
rejected ballots as other precincts. (New York
Times, 11/12/01)
9
Design is important in a lot of things
including assessments! http//go.to/funpic
10
Assessments Discussed in this Training

• English Language Proficiency Listening/ Speaking
  Test - Computer Based
• English Language Proficiency Reading and Writing
  Test - Paper/Pencil
• Grade level Academic Content Area Assessment -
  Computer Based
• Grade level Academic Content Area Assessment -
  Paper/Pencil

11
Universally Designed Assessments Can Increase
Reliability and Validity of Assessments for
English Language Learners

• Analyses of large data files (Abedi, 2002) found
  language as a source of measurement error that
  can negatively impact reliability of test results
  for ELLs.
• Language has also been found to be a source of
  construct irrelevance that can have a negative
  impact on the validity of the results for ELLs
  (Abedi, 2002).

12
Universally Designed Assessments Increase
Usability for the End Users STUDENTS!

• In the past, emphasis has been placed on
  developing test items without consideration of
  the students who are the ultimate end users
• The students who are tested have changed and are
  no longer even remotely comparable to the
  designer in skill set, aptitude, expectation, or
  in almost any attribute that is relevant to the
  design process (Rubin, 1994)

13
Student Characteristics
Just one of many reasons that we need to be
talking about universally designed assessments!
14
Complete these well-known phrases

1. A penny saved is . . .
2. An idle mind is . . .
3. Dont bite the hand that . . .
4. Its always darkest before . . .
5. Strike while the . . .
6. Twos company, threes . . .
7. If at first you dont succeed, . . .

15
Heres what the kids say!
16
A penny saved is . . .
. . . not much.
17
An idle mind is . . .
. . . the best way to relax.
18
Dont bite the hand that . . .
. . . looks dirty.
19
Its always darkest before . . .
. . . Daylight Savings Time.
20
Strike while the . . .
. . . bug is close.
21
Twos company, threes . . .
. . . the Musketeers.
22
If at first you dont succeed, . . .
. . . get batteries.
23
What seems obvious and clear to test developers
may not be all that obvious and clear to
students.

• It takes consistent effort and guidance to make
  sure that test items and tests really are
  accessible to all students.

24
This is a BIG DEAL!

• Requirements for universally designed assessments
  are showing up in federal legislation

25
Title I Regulations introduced the need for
universally designed assessments
Assessments must be designed to be accessible
and valid with respect to the widest possible
range of students, including students with
disabilities and students with limited English
proficiency.
Sec. 200.2(b)(2)
26
Increasingly, we need to think about students
with a variety of different language and cultural
background characteristics
27
The number of English Language Learners in
schools has increased dramatically

• Nearly 4 million LEP students in grades K-12 in
  2001-2002 (71.9 increase from 1991-1992)
• Represent about 8.4 of all public school
  students in the U.S.
• Enrolled in almost half of all public schools
  nationwide

28
The number of ELLs with special needs is
increasing too!

• Most recent estimate is 9 of the total ELL
  population is in Special Education
• Estimated at about 357,000 students
• More than 50 of ELLs with disabilities
  classified as learning disabled

29
The bill for the federal reauthorization of the
Individuals with Disabilities Education Act
(IDEA) also includes universal design

• UNIVERSAL DESIGN- The State educational agency
  (or, in the case of a district-wide assessment,
  the local educational agency) shall, to the
  extent possible, use universal design principles
  in developing and administering any assessments.

Sec 612, Senate Bill 1248
30
So, when we think about all students, we have to
think about students who have a variety of
cultural, physical, sensory, and processing
characteristics.
31
Caution
While universally designed assessments can make
tests more equitable, producing results that are
more valid for all students, they cannot replace
instructional opportunity!
32
Elements of Universally Designed Assessments

• Inclusive assessment population
• Measures what it intends to measure
• Respects the diversity of the assessment
  population
• Clear format for text

33
Elements of Universally Designed Assessments

• Clear pictures and graphics
• Concise and readable text
• Amenable to accommodations

34
Inclusive Assessment Population

• Universally designed assessments

• Consider all types of students in the general
  curriculum from the beginning
• Include English language learners and students
  with disabilities in item tryouts and field
  testing

35
Example of Field Test Specifications

• The statewide field test will include students
  with disabilities and English Language Learners
  using the accommodations they would normally use
  in the large-scale assessment.
• There may be need for over sampling of these
  populations to ensure a valid field test.

36
Measures What it Intends to Measure

• Universally designed assessments reflect good
  measurement qualities

• Reflects the intended content standard (reviewers
  have information about what is being measured)
• Minimize skills required beyond those being
  measured

37
Suppose a test item requires a student to read an
Aesops fable

• What could this item measure?
• Decode text?
• Comprehend extended passages?
• Extent of vocabulary?
• Understand the moral or point of the fable?
• Discuss the common elements of any fable?
• Compare and contrast fables with news reports?
• Articulate the relationship between the fable and
  the overall culture?
• Anything else? (National Center on Accessing the
  Curriculum, 2003)

38
Suggestions by Researchers . . .

• Popham (2001) suggests creating an assessment
  description for each item that concretely
  describes what is being tested.
• Chiu Pearson (1999) suggest determining
  non-construct behaviors needed to complete task.

39
Respects the Diversity of the Assessment
Population

• Accessible to test takers (consider age,
  gender, ethnicity, disability, socio-economic
  level)
• Avoid content that might unfairly advantage or
  disadvantage any student subgroup

40
According to the National Research Council
(1999), bias arises when
Deficiencies in the test itself result in
different meanings for scores earned by members
of different identifiable subgroups.
41
Consider the Effect of Cultural Bias
A. B.
A.
B.
If you measure from the nose to the tail, how
much longer is lizard B than lizard A? Use the
ruler provided in your testing kit. Give the
answer in centimeters.
42
Experience can be crucial!

• To raise money for a trip to the Wolfridge
  Environmental Learning Center, sixth graders at
  Johnson Middle School are selling raffle tickets.
  The raffle prize is an electric scooter worth
  more than 300. A total of 500 tickets were
  sold. You bought two raffle tickets, your sister
  bought three and your father bought one. What is
  the probability that someone in your family will
  win the prize?

43
Clear Format for Text

• Universally designed assessments use text that
  can be read quickly and effortlessly

• Standard typeface
• Type size at least 12 point
• Wide spacing
• High contrast
• Sufficient leading (Spacing) between lines of
  text
• Staggered right margins (no right justification)
• No background graphics behind text

44
Typeface Standard typeface, with upper and
lower case letters, is better than italic, small
caps, or all caps
Adapted from Making Text Legible by A. Arditi
45
Type Size
All type should be at least 12 point (including
captions, keys, labels, and footnotes). Remember
that point size differs among typefaces.
Adapted from Making Text Legible by A. Arditi
46
Spacing Space between letters and between words
is wide
Adapted from Making Text Legible by A. Arditi
47
Contrast
Text should be printed with the highest possible
contrast.
Adapted from Making Text Legible by A. Arditi
48
Leading Spacing Between Lines of Text
Adapted from Making Text Legible by A. Arditi
49
Staggered Right MarginsText is easier to read
when it is justified on the left and unjustified
on the right. This prevents pockets of spaces.

• Text is easier to read when it is justified
  on the left and unjustified on the right. This
  prevents pockets of spaces.

(APH, 2002) Used with permission
50
Text with background graphics or other media is
difficult to read.
This is a picture of a starfish in- side a
vortex. Of course to you it might look like a
giant flower or an octopus in a whirlpool. You
might even think it is op- art or a computer
rendering of a bad dream. However this photo is
to demonstrate to you how difficult it is to
read text when it is placed over a busy
background.
(APH, 2002) Used with Permission
51
Clear Pictures and Graphics

• Clear, non-fuzzy pictures
• Dark lines (minimum use of gray scale and
  shading)
• Sufficient contrast between colors
• Color is not relied on to convey important
  information or distinctions
• Label pictures and graphics when possible

52

• Universally designed assessments use non-text
  materials just as carefully as text materials

• Only essential illustrations are used (referred
  to in text and necessary to answer question)
• Illustrations for interest may draw attention
  away from content being assessed

53
Is this border distracting? Is it necessary?
54
Fence or Bridge or What?
fence
Add context and labels to help identify pictures
55
Is the swimmer at the bottom of the page
distracting?
Is this a better way to remind test takers to
continue?
56
Concise and Readable Text

• Commonly used words
• Vocabulary appropriate for grade level
• Minimum use of unnecessary words
• Idioms avoided unless idiomatic speech is being
  measured
• Technical terms and abbreviations avoided (or
  defined) if not related to the content being
  measured
• Sentence complexity is appropriate for grade
  level
• Question to be answered is clearly identifiable

57
Clear Instructions

• Ensure that test takers can respond to a task in
  the manner that the test developer intended
  (AERA, APA, NCME, 1999).
• While writers might think certain expectations
  are obvious, if they are not explicit in the
  item, then they are subject to honest
  misinterpretation in the responses (Kopriva,
  2000, p. 39).

58
Assessments designed to better include English
language learners benefit all types of students!

• Students have the experience to understand the
  items
• Language is clear, simple and indicates precisely
  what is required from student (Plain language)
• Questions are amenable to supports that ELLs
  might use
• Cognitive demands are reasonable

59
Abedi research suggests that linguistic
complexity of test items is a significant source
of measurement error for English language
learners (and students with disabilities)
Examples of Linguistic Modifications
Familiarity or frequency of non-math vocabulary
(unfamiliar or infrequent words changed)
Tetris
Video game
Length of nominals (long nominals shortened)
Last years class vice president
The vice president
(Abedi, et al., 2001)
60
Linguistic Modifications (continued)
Question phrases (complex question phrases
changed to simple question words)
At which of the following times
When
Voice of the verb phrase (passive verb forms
changed to active)
If a marble is taken from the bag
If you take a marble from the bag
Relative clauses (relative clauses removed or
recast)
A report that contains 64 sheets of paper
He needs 64 sheets of paper for each report
61
Linguistic Modifications (continued)
Conditional clauses (conditionals either replaced
with separate sentences or order of conditional
and main clause changed)
If Lee delivers x newspapers
Lee delivers x newspapers
Abedi found that English language learners who
received a linguistically modified version of the
math test performed significantly better than
those receiving the original test items. They
also performed significantly faster.
62
Over 80 of the students also told Abedi they
preferred the modified math test because

• Its easier to read, and it gets to the point, so
  you wont have to waste time
• Its shorter and doesnt have, like, complicated
  words
• I might have a faster time completing that one
  cuz theres less reading

(Abedi, et al., 2001)
63
Sample Readability Guidelines

• Use simple, clear, commonly used words,
  eliminating any unnecessary words (Abedi, 2002)
• Clearly define any technical terms that are used
• Break compound complex sentences into several
  short sentences. State the most important ideas
  first (Rakow Gee, 1987)
• Introduce one idea, fact, or process at a time
  (Rakow Gee, 1987)

64
Recommendations to Improve Accessibility of Text

• Simple, brief and consistent sentence structure
  in items
• Consistent and clear paragraph structure
• Present tense and active voice
• Minimal paraphrasing and rewording. If used,
  identify the original statement in parentheses

From Kopriva (2000)
65
Recommendations to Improve Accessibility of Text

• Minimal use of pronouns. Follow a pronoun with
  the term it refers to in parentheses
• High frequency words
• Avoid words with double meanings and
  colloquialisms. If used, define them in the text.

From Kopriva (2000)
66
Example Reading Passage Newspaper Excerpt

• Para. 1 When Nicole Zachor, Laura Swanson and
  Carol Hinz started work on a project for history
  class a few months ago, the White Bear Lake
  sophomores had no idea that it would be displayed
  at the Smithsonian Institutions National Museum
  of American History
• Para. 12 This year junior and senior high
  students started work on their projects in
  January or February. A project can be a
  research paper but it can also be a group of
  individual media presentation, display
  presentation or performance

67
Side-by-Side Translation

• Which of the following is a condition for a
  student to participate in National History Day?
• The student must be a junior or senior in high
  school.
• The student must be able to go to Washington,
  D.C.
• C. The student must do a project related to
  the national topic.
• D. The student must do the project on his or
  her own by himself or herself.

• Cuál de las siguientes es una condicíon para que
  un estudiante participe en el Día Nacional de
  Historia?
• Los estudiantes deben estar en grados once o doce
  de High School.
• El estudiante debe estar en posibilidad de ir a
  Washington, D.C
• El estudiante debe hacer un proyecto relacionado
  a un tema nacional.
• El estudiante debe hacer el proyecto por sí solo.

68
Ordering Pizza (Original Item)
The cafeteria manager surveyed the students in a
middle school to find out if they would buy Brand
X pizza on Friday if the manager sold it. She
made a circle graph to display the results of her
survey.
NO
YES
Based on the results of the survey, answer the
following questions

1. What fraction of students would buy Brand X pizza
   on Friday?
2. What percent of students would buy Brand X pizza
   on Friday?
3. There are 1200 students in this school. How many
   students will buy Brand X pizza on Friday if the
   managers survey is accurate?

69
Ordering Pizza (Revised Item)
Maria surveyed the students in her school to find
out if they would like pizza on Friday. She made
a circle graph to display the results of her
survey.
NO
YES

1. What fraction of students said yes?
2. What percent of students said yes?
3. There are 1200 students in Marias school. How
   many students said yes?

(From Hanson, et al., 1998)
70
Cognitive Demands

• Amount of text not relevant to items
• Length of text
• Number of long texts
• Timing (may be unspoken)
• Number of unfamiliar words
• Placement of definitions (in text, to side,
  separate) if allowed
• Location of native language text if provided

71
Amenable to Accommodations

• Universally designed assessments allow needed
  accommodations to be used

• Plan for students who continue to need
  accommodations
• Facilitate the use of accommodations such as
  bilingual dictionaries or translations (when
  appropriate), assistive technology, oral
  presentation, large print, Braille

72
What accommodations might be difficult with this
item? Could this item be revised without
knowing what it is intended to measure?
73
Here is a revised graphic that would clarify the
item.
Items and revision used by permission from the
state of NH and DRC
74

• Universally designed assessments consider the
  design of the response venue as well as the
  assessment itself

• Large bubbles that avoid most challenges of low
  vision or difficulty with fine motor skills
  (Grise, Beattie, Algozzine, 1982)
• Consideration of age of students in selecting
  format (avoid separate answer sheets for younger
  students) (Tindal and others suggest 4th grade as
  transition point).

75
Clear Response Grid
Gray columns changed to yellow. Larger
bubbles. Easier to see.
Hard to see
(APH, 2003) Used with permission
76
Computer-Based Testing All of the Elements of
Universal Design Plus Some Additional
Opportunities and Challenges
77
Opportunities

• Efficient administration
• Preferred by students
• Improved writing performance
• Built-in accommodations
• Immediate results
• Efficient item development
• Increased authenticity

78
Challenges

• Use of technology cannot take the place of
  content mastery
• Issues of equity and skill in computer use
• Added challenges for some students
• Technological challenges
• Security of online data
• Lack of expertise in designing accessible Web
  pages
• Prohibitive development cost

79
Example Kentucky Online Assessment

• Web-based, individualized assessment
• Students with IEP or 504 Plan that specifies need
  for "reader" as an instructional and assessment
  accommodation
• Students who require and routinely use
  text-reader or screen-reader technologies to
  access printed material in classroom instruction
  and assessment
• Students who have accessed and used the CATS
  Online Practice Area.
• Based upon success of pilot studies, 16
  districts, 31 Schools, 204 students
  participated in live CATS Online in the spring
  of 2003

80
CATS Online
81
How to Develop Universally Designed Assessments

• Develop frameworks for alignment of items with
  content
• Include elements of universally designed
  assessments in test specifications
• Develop guidelines for item developers to follow

82
How to Develop Universally Designed Assessments

• Flag items with bias and design issues
• Consult with end users students representing
  all subgroups
• Consult with bias or sensitivity review teams
  representing all subgroups
• Analyze field test data and statewide test
  results by item and subgroup

83
What Reviewers Need to Know

• Purpose of the test
• Standard tested by each item
• Description of test takers (e.g. age, geographic
  region)
• Test instructions
• Overall test and response formats
• Use of technology
• Field test results by item and subgroup (when
  completed)

84
Review Team Members

• Grade-level content experts
• Representatives of major cultural groups
• Representatives of major disability groups
• Second language acquisition experts
• Who else?

85
Remember!

• This is a work in progress
• Considerations put important issues on the table
• Ongoing input is important!

86
Item Review Materials

• Considerations and notes
• Research base for considerations
• References
• Considerations form for item reviewers
• Sample revised items

87
NCEO Resources
Visit education.umn.edu/nceo or Search for
NCEO Click on Universal Design