Disproportionality and Best practices in Unbiased Assessment

1
Dis-proportionality and Best practices in
Un-biased Assessment

• Steve Hirsch, PhD, NCSP
• WSU EWU School Psychology program coordinator
• East Valley, School Psychologist
• smhirsch_at_wsu.edu

2
Best Realistic Practice in Non-Discriminatory
Assessment

• Completely unbiased assessment is an illusion.
• Non-discriminatory assessment is not a search
  for an unbiased test but rather a process that
  ensures every individual, not just those who are
  deficit in some way, is evaluated in the least
  discriminatory manner possible. Ortiz, 2004

3
Best Realistic Practice in Non-Discriminatory
Assessment

• What is needed is a comprehensive, systematic
  framework comprising a broad range of methods and
  procedures.
• But just as I am ready to dismiss standardized
  testing as invalid, I will present an intriguing
  model that is logical and allows for the use of
  standardized tests of cognitive functioning

4
Workshop Itinerary

• The concept of dis-proportionality
• The etiology of the concept
• Cultural and Linguistic competency
• Inadequacies of standardized tests
• Value of formative assessment
• Assessment of the Learning Ecology-FAAB
• Our traditional attempts to reduce bias
• The matrix of cultural/linguistic factor loading
  in cognitive assessment

5

• Dis-proportionality only exists in the soft
  handicapping categories

(hard) Deafness Blindness Orthopedic
impairment Severe mental retardation
(soft) Specific learning disability Mental
retardation Emotional disturbance
6
In Washington
Enrollment
of pop. in SpEd
Child Count
2.7 4.0 16.4
Am Ind /AN 8.7 4.7
6 Asian/PI 5.7
7.7 14.8 Black 14.6
14.6 10.6
Hispanic 65.8 68.9
10.7 White
7
(No Transcript)
8
(No Transcript)
9
State Totals
10
Washington State
Summary Adequate Yearly
Progress 2007-08
11
Class of 2008 Washington State
Dropout Rates

12
Doubt is uncomfortable, certainty is ridiculous.

-Voltaire (François-Marie Arouet)
13
Where are we going wrong? From whence comes
dis-proportionality?

• Assessment/identification process?
• Inaccessibility of the general curriculum to some
  ethnic minorities?
• Is it a result of pre-conceived stereotypes?
• Could it be that some populations are actually
  more prone to the deficits in the psychological
  processes that define a learning disability (i.e.
  memory, attention, processing speed, language,
  motor)?

14
Debate How is dis-proportionality in the
category SLD occurring?

• Given
• The most dis-proportionate populations have FS
  average scores significantly below whites on IQ
  testing (the tests are biased that way)
• To be identified as LD you must have a
  significant discrepancy between IQ and
  achievement on standardized test
• We all know that it is nearly impossible to
  qualify a child with borderline to low average IQ
• How then are we qualifying a disproportionate
  number of blacks and Native Americans as LD?
• Professional Judgment?

15
How did we get to this point? Could
dis-proportionality be a result of how we view
special education?

• From WAC, special education represents
  brokenness of the nervous system. Keep em
  out of sped
• To the ELL parent, special education may
  represent what it was in the previous culture.
  keep em out
• To an advocacy group, special education often
  represents a label, stigma but protection in form
  of IEP keep em out unless they qualify
• To the team, special education often represents
  additional help usually in a small group or
  individual setting. E.g. tutoring. put them in

16
Could dis-proportionality be a result of how we
view special education?

• If special education is viewed as a way of
  helping a child using resources not available to
  the general education community, or is viewed as
  protection in form of IEP, then how could we not
  expect certain minority populations to be
  over-represented?
• They dont test academically as well on our
  standardized achievement tests and therefore are
  more likely to look as if they need extra help.

17
Best Realistic Practice in Non-Discriminatory
Assessmentala APA

• Consider influence of language and culture on
  behavior
• Consider validity of methods and procedures
• Make interpretations of data within context of a
  students linguistic cultural characteristics
• When in doubt, hide behind vagueness

18
Needed Cultural Linguistic Competency

• Cultural competence- knowledge base of, or direct
  experience (with values, attitude, beliefs and
  customs of particular culture) that will guide
  data collection analysis.
• Doesnt come from a book or taking a trip or even
  looking out your kitchen window at another
  country. Look for professional development or
  experiences with community or advocates

19
Willingness To see and Accept help
Family/school
Acculturation indicators
Perception of Disability
Discipline
20
Needed Cultural Linguistic Competency

• Linguistic competence Ability to communicate
  effectively in individuals native language (no
  interpreter)---i.e. multi-lingual
• Typically not reality so next best thing
• Possession of a knowledge base related to first
  and second language development and instructional
  methodology as it relates to ELL

21
And if you are going to use standardized
tests-skills you need to know

• Adequacy of sampling norms
• Full range of abilities being measured in
  subtests
• Linguistic demands and cultural loading
• Reliability and validity of cultural-fair tests
• Sources of potential bias

22
And if you are going to use standardized
tests-keep in mind that

• Published tests of cognitive ability are often
  viewed as psychometrically non-biased due to
  sampling norm that mirrors census data.
• Acculturation impacts appropriateness of tests
• Decisions must be made on individual basis

23
Debate Is a particular cultural group ever
adequately represented in a normative sample?

• If 3 of test sample represents a group that
  constitutes 3 of the population, does that mean
  that the test is appropriate for your particular
  student?
• The 3 in test sample-to what extent are they
  representative of their cultural population?
• To what extent is your student representative of
  their cultural/linguistic population?
• Just because a culture is represented in a sample
  doesnt mean that the test is fair to the culture
  

24
Adding insult to injury

• If you look at sampling norm data in test
  manuals, you typically find the following
  (TONI-3, 1997)
• Carefully stratified across US geographic regions
  (northeast midwest south west-census matched)
• Ethnicity categories- African American Hispanic
  Asian Native American.
• While they are careful not to conclude that a
  mid-westerner is similar to a westerner, they
  assume that Chinese, Vietnamese, Korean, Hmong,
  Indonesian, Filippino, Samoan, Indian, Japanese
  and Hawaiian among hundreds of other cultures are
  all identical

25
Other important issues

• Degree to which the test is linguistically loaded
• Limited verbal load (WISC perceptual
  organization)
• Non-verbal Minimal verbal interaction (KABC)
• Language-free (UNIT)
• Degree to which a test, even if language-free, is
  culturally loaded
• Bottom line we know very little how different
  bilingual populations perform on tests normed on
  children from monolingual single-culture
  environment

26
What assessment tools are available, and what
could be assessed Cognitive

• Reduced Language Load (WISC-PO index)
• Would you want the world to believe that your IQ
  is solely a function of non-verbal subtests? Not
  me!
• Non-verbal assessments (KABC-II DAS)
• If instructions frame problem-solving strategy
  but instructions are in English
• Language-free assessment (UNIT)
• But cognitive functioning is in large part, verbal

27
What assessment tools are available, and what
could be assessed Academic

• Translated assessments e.g. Woodcock-Munoz
• Problem is that bilingual children may never
  received formal instruction in their native
  language. Academic achievement must be viewed
  carefully
• Curriculum Based Measurement (ORF) when used with
  ELL students
• Has been shown to have good reliability
• Has been shown to have good validity
• Has been shown to be a reliable progress monitor

28
Academic Assessment of ELL Life is good-its
almost time for Gonzaga basketball epiphany

• Global norm-referenced tests invalid
• Translated versions of tests-questionable
• ORF susceptible to word-calling
• What if we looked at more than the correct number
  of words read per minute during DIBELS? What if
  we focused on a composite of speed, accuracy and
  prosody? True Oral Reading Fluency. Will it
  correlate with overall reading or WASL reading?
  Do ELL students improve in prosody? Need
  research!

29
Academic Assessment of ELL fear of shoveling 10
ft of snow epiphany

• We have all sorts of benchmarks for ORF but none
  reflect the ELL student. Are there unique
  learning trajectories in ORF for ELL students
  based on their English language proficiency? But
  how does acculturation degree factor into this?

30
What assessment tools are available, and what
could be assessed Instructional Environment

• Do students have sufficient background to
  understand content of instructional materials?
• Can students relate to content of instructional
  materials from cultural perspective?
• Is curriculum being adapted to needs of bilingual
  students?

31
What assessment tools are available, and what
could be assessed Instructional Environment

• A difficult task Evaluate teacher expectations
  of bilingual students and how it might impact
  instruction?
• Do we have adequate benchmarks to assess the
  progress of ELL students against?
• Interesting study for someone out there How are
  teachers expectations of a student influenced by
  students cultural background?

32
Do teachers have different expectations of
children of differing cultural backgrounds?

• Based on your interactions with children of these
  cultural backgrounds, rank them in terms of the
  likelihood of this child returning homework daily
• ___Russian
• ___Hmong
• ___Samoan
• ___Vietnamese
• ___Hispanic
• ___Chinese
• ___Native American

33
What assessment tools are available, and what
could be assessed Instructional Environment

• The Real Danger Shared 40 yrs ago
• Standardized tests label black children as less
  educable place black children in special
  classes perpetuate inferior education assign
  black children to lower education tracks than
  whites deny black children higher educational
  opportunities and destroy positive intellectual
  growth and development of black children.

34
Are standardized tests really that poor for ELL
students?

• Norms usually limited to small samples of
  minority children
• Norms routinely exclude students with limited
  English proficiency
• Test items tap info that minority children may
  not be familiar with
• Tests typically dont allow examiners to probe
  and inquire further
• Correct items on tests usually based on majority
  childrens responses
• Standardized testing procedures assume that
  students have appropriate test-taking skills

35
What about translated tests?

• Not really the answer
• Sometimes vocabulary and concepts not easily
  translated (e.g. Yiddish word, mensch not
  easily translated to English)
• Difference in dialects result in translation
  errors
• Some concepts change meaning once translated

36
So if not standardized tests, what ?

• Then we will look at Best Practices in
  non-discriminatory assessment- A comprehensive
  framework

37

• For every complex problem there is an answer
  that is clear, simple, and wrong.
• H. L. Mencken (1880-1956)

38
Comprehensive Framework

• Evaluate the learning ecology
• Evaluate language proficiency
• Evaluate opportunity for learning
• Evaluate educationally relevant cultural and
  linguistic factors
• Develop, and revise hypotheses

39
Assessment of learning ecology

• Exploration of extrinsic factors that might lead
  to difficulties
• Functional Assessment of Academic Behavior (FAAB)
• Interviews w/parent, teacher, child
• Observations in multiple settings
• Assess home, classroom teacher/student
  interaction home/school interaction

40
Assessment of opportunity for learning-specific
factors to be examined

• Regularity of attendance
• Experience with school environment
• Language match (native vs. English)
• Parents ability to support language instruction
• Years of instruction in English

41
Assessment of opportunity for learning-specific
factors to be examined

• Quality of ESL instruction or instruction in
  native language
• Frequency of school change
• School attitude expectations re. dual language
  learners
• Socialization vs. isolation from peers

42
Assessment of educationally relevant cultural and
linguistic factors

• Learning takes place everywhere, not just school.
  Many factors outside educational setting can
  affect the learning process.
• Exposure to two or more cultures or languages
  during early childhood may create circumstances
  that cause child to have very different
  experiences than other children.

43
Evaluate and revise hypotheses

• Only when enough confidence exists in the belief
  that there are no plausible external factors that
  could account for students learning difficulties
  is it time to consider internal factors.
• There should also be evidence that efforts to
  reduce impact of external factors through
  intervention were unsuccessful.

44
So far

• The above steps should all be part of a
  pre-referral process not part of the special
  education assessment

45
Reduce bias in traditional testing practice

• Bias is due to non-representational sampling- key
  is acculturation and English proficiency. Even
  with native language tests, bias remains. Two
  traditional approaches to reduce bias

46
Traditional approaches to reduce bias

• Administer tests in standardized way-attempt to
  evaluate results in non-discriminatory manner
• Use of existing locally developed norms-more
  appropriate comparison group. Standardization
  then helps for comparison
• Deviations from standardization would produce
  results that are unknown and unpredictable as
  does translated tests
• Test selection based on properties relating to
  cultural loading

47
Traditional approaches to reduce bias

• Modify testing process in a way that is less
  discriminating
• Any modification will of course violate
  standardization and impact validity and
  interpretability of results
• You also lose ability to use results for
  comparison purposes
• We typically dont know impact of any
  modification
• We could look at data qualitatively

48
Traditional approaches to reduce bias some
modifications

• Bilingual administration
• Use of extended-expanded instructions on sample
  items
• Mediation of concepts to ensure comprehension
  prior to test administration
• Repetition of items to facilitate comprehension

49
Traditional approaches to reduce bias some
modifications

• Extension or elimination of time limits
• Acceptance of alternate responses e.g. non-verbal
  gestures
• Additional probing and querying of incorrect or
  partially correct answers

50
Utilize authentic assessment procedures

• Whereas standardized norm-referenced tests are
  driven by questions and needs related to
  classification, diagnosis and legal eligibility,
  authentic assessment is geared more towards
  answering questions regarding instructional needs
  and interventions.

51
Evaluate data within context of learning ecology

• Interpret and analyze in terms of acculturation
  and English language proficiency-cohort analysis
  through OSPI
• Knowledge of factors that may have played a part
  in creating significant differences between the
  experience of the individual in terms of
  acculturation or language development- provides
  the least discriminatory framework with which to
  evaluate.

52
Evaluate data within context of learning ecology

• Care should be taken not to give too much
  importance to any single piece of information.
  There is a tendency to jump on any data that is
  inspirational in that it was unexpected.

53
Linking Assessment to Intervention

• Assessment and evaluation are not interventions!
• Problems do not disappear as a result of
  evaluation!
• Assessment is of little value without appropriate
  interventions and treatment options. Does the IQ
  really yield valuable info re. intervention?
• The non-discriminatory assessment yields data
  ripe for intervention

54
Is this leading up to RTI? You betcha! (Palin,
2008)

• The process is the same for the bilingual student
  as any student
• Identify the at-risk student. Not all bilingual
  students are at-risk
• Use diagnostic assessment (incl. FAAB) to
  determine appropriate interventions. Not all
  bilingual students need the same interventions
• Monitor progress during intervention
• Evaluate effectiveness of intervention and modify
  intervention if necessary

55
Drop out rates
Graduation rates
Proficiency on State assessments
RTI
Academic Achievement
Discipline
LRE
Disproportionate representation in special ed
56
Assessment under RTI

• CBMs are probably the most efficient and maybe
  the most valid tool for identifying the at-risk
  bilingual student
• In reading- ORF(e.g. DIBELS) Aimsweb
• In math- Aimsweb, MathAlert! Or MAP
• In writing- 1min think/3 min write

57
Assessment under RTI

• Diagnostic evaluation for bilingual students
  under RTI can include
• FAAB for learning ecology
• Reading QRI DRA BRI assess five essential
  reading components
• Math KeyMath MAP
• Written language in response to prompt assess
  seven essential elements

58
Essential Reading and Writing Components

• Reading phonemic awareness phonological
  processing fluency vocabulary comprehension
• Writing
• Fluency (total words written)
• Legibility (letter formation analysis)
• Conventions (spelling/punctuation/capitalization)
• Syntactic Maturity (varied sentence
  lengths/types)
• Semantic Maturity (variety of words/vocabulary)
• Content (organization, cohesion)
• Writing Process( plans ahead, transitions)

59
Where does this all leave you today? In my opinion

• No valid standardized test
• Assessment a framework or process, not a test
• Assess the learning or instructional environment
• Knowledge about the specific culture
• Knowledge about bilingual education
• Typical learning progress rates of ELL students

60
drat

• Just when I thought my case was airtight and my
  message was clear

61
A model for the cognitive assessment of ELL and
culturally diverse students that makes sense

• Subtests on the various cognitive functioning
  tests (e.g. WISC IV WJIII) can be rated on their
  sensitivity to either linguistic or cultural
  diversity.
• see the NASP publication Ortiz (2004)
  Comprehensive Assessment of Culturally and
  Linguistically Diverse Students

62
Comprehensive Assessment of Culturally and
Linguistically Diverse Students

• Examining the pattern of scores on the subtests,
  taking into account this sensitivity, might allow
  us to discriminate between
• cognitive difficulties resulting from a lack of
  language/cultural experience or something else
  (i.e. disability)

63
Matrix of Cultural Loading and Linguistic
Demand(WISC IV)
Dependency on linguistic competency
Low Medium High

Dependency on cultural competency
High Medium Low
64
And is all this realistic?

• Just my preaching!
• Make this a professional goal for the next year
• WSASP will provide professional dev. opportunity
• Familiarize yourself with the cultures of your
  community-attend cultural activities church etc.
  
• What is realistic is to learn more about the FAAB
  

65
It is easier to build strong children than to
repair broken men.
Frederick Douglass