Lisa Harris,

1
Comparison of Student Performance Between Teacher
Read and CD-ROM Delivered Modes of Test
Administration of English Language Arts Tests

• Lisa Harris,
• University of South Carolina

2
Background

• NCLB, IDEIA, Section 504 of Rehabilitation Act
• Computer-based accommodations available on
  national tests
• AP exams
• SAT
• GRE
• GMAT, LSAT
• Increased computer availability in schools
• Increased use of computer accommodations on
  state-wide accountability tests

3
Past Findings Where do we go from here?

• Read aloud administration
• Accommodation or modification?
• Other issues
• Cuing
• Pacing
• Computer skills
• Reading Level
• Item type

4
Past Findings

• Read-aloud Accommodation Delivery Options
• Live reader with written script
• Cassette/CD with cassette/CD player
• CD played on computer with test booklet
• Video taped reader (head shot)
• Video taped reader, only text displayed
• Video tape with head shot and text
• Test on computer with computer generated voice
• Test on computer with human voice

5
Call for more research

• Studies using math tests
• Possibly confounded by lack of content
  knowledge(Crawford Tindal, 2004 Bolt
  Thurlow, 2006)
• Few studies compare delivery methods
• Call for more research on read-aloud delivery via
  computer (Miranda, 2004 Calhoon, 2000)
• 1999 Standards for Educational and Psychological
  Testing
• Support should be provided for any assertion
  that scores obtained using different items or
  testing materials, or different testing
  procedures, are interchangeable for some
  purposes AERA, 1999, p.57

6
Research Questions

• Does the English Language Arts (ELA) test have
  the same factorial structure between oral script
  and CD-ROM modes of test administration?
• Controlling for prior ELA performance, are there
  differences in student performance between test
  administration modes?
• Is there an interaction between test
  administration mode and student disability?
• Does student performance vary between the oral
  script and CD-ROM modes of test administration?
• How do the above results hold across grade
  levels?

7
Methods Instruments

• Palmetto Achievement Challenge Test (PACT)
• Part of the state accountability system
• Based on state academic content standards
• Grades 3-8
• ELA, math, science and social studies
• CDs available in grades 5-8

8
Delivery Methods CD-ROM

• Follow along in test booklet
• Record answers in test booklet
• No passages or questions appear on screen
• Human voice

9
CD Screen Shoots
10
CD Screen Shoots
11
CD Screen Shoots
12
OS Delivery

• Teacher-reader follows a script.

13
Methods Instruments

• Depth of Knowledge Rubric for Selected-Response
  Items
• Development
• Description
• DOK1 Verbatim recall and simple understanding
• DOK2Basic reasoning skills, simple extension
  beyond what is explicitly stated
• DOK3Complex reasoning
• DOK4Extended reasoning, inference, planning.
• Implementation

14
DOK Level Examples

• DOK1 Questions focus on verbatim recall and
  simple understanding. Questions are related to
  parts of a text rather than the text as a whole.
  
• According to the poem, how has the speaker
  already tried to solve the noise problem?
• by telling the mother
• by asking Ray to whisper
• by giving Ray earplugs
• by shutting the bedroom door

15
DOK Level Examples

• DOK2 Require basic reasoning skills,
  comprehension on the literal level, and simple
  extension beyond what is explicitly stated.
  Questions require some mental processing of the
  text or portions of the text.
• What is the main idea of this passage?
• A. Louis suffered from poor health.  
• B. Louis could have been a great painter.
• C. Louis wrote books that many children enjoy.
• D. Louis used his talents throughout his life.

16
DOK Level Examples

• DOK3 Questions require complex reasoning.
  Questions require an understanding of the text as
  a whole.
• When the poet says Like medals with their
  ribbons frayed and wavering (lines 6162), she
  is referring to
• A)  victory
• B)  fishhooks
• C)  trophies
• D)  fish scales

17
DOK Level Examples

• DOK4 Questions require extended reasoning,
  inference or planning. Questions go beyond the
  literal text and require a deep, purposeful
  understanding of the text as a whole and/or and
  understanding of the text as a whole in relation
  to other texts.
• Which word best describes the tone of The Long
  Hill?
• A. fearfulness
• B. encouragement
• C. wishfulness
• D. disappointment

18
Methods Participants

• Students in grades 6-8 who took ELA PACT in
  spring 2007 using the read-aloud administration

Oral Script CD-ROM
Total 4966 869
Grade 6 1681 259
Grade 7 1753 310
Grade 8 1532 300
19
Methods Participants

• Due to small sample size for non-LD students
  disability categories were collapsed into the
  following
• Category 1 students with learning disabilities
• Category 2 all other disabilities including
  speech/ language impaired, deaf or hard of
  hearing, blind and visually impaired,
  orthopedically impaired, autistic, emotionally
  disabled, educable mentally disabled, other
  health impaired, traumatic brain injury, and
  multiple-disabled due to small sample size

20
Demographic Characteristics of the Students Grade
6
Demographic OSn 1681 OSn 1681 CD-ROMn 259 CD-ROMn 259
Characteristic n n
Gender
Female 511 30.0 91 35.0
Male 1170 70.0 168 65.0
Ethnicity
African American 890 53.0 115 44.0
Hispanic 44 2.6 14 5.4
White 719 43.0 123 47.5
Eligible for free or reduced lunch 1391 82.7 192 74.1
IEP Labels
Speech/Language 270 13.4 32 10.8
Learning Disabled 1180 58.8 194 65.8
Educable Mentally Disabled 289 14.4 33 11.2
Other Health Impaired 125 6.2 25 8.5
Disability Category
1 (Learning Disability) 1034 61.5 169 65.3
2 (All others) 647 38.5 90 34.7
21
Demographic Characteristics of the Students Grade
7
Demographic OSn 1753 OSn 1753 CD-ROMn 310 CD-ROMn 310
Characteristic n n
Gender
Female 543 30.0 86 28.0
Male 1210 69.0 224 72.0
Ethnicity
African American 1027 58.6 154 49.7
Hispanic 44 2.5 8 2.6
White 661 37.7 136 43.9
Eligible for free or reduced lunch 1444 82.5 242 78.1
IEP Labels
Speech/Language 209 10.4 32 9.3
Learning Disabled 1233 61.1 212 61.5
Educable Mentally Disabled 294 14.6 51 14.8
Other Health Impaired 142 7.0 29 8.4
Disability Category
1 (Learning Disability) 1095 62.5 198 63.9
2 (All others) 658 37.5 112 36.1
22
Demographic Characteristics of the Students Grade
8
Demographic OSn 1532 OSn 1532 CD-ROMn 300 CD-ROMn 300
Characteristic n n
Gender
Female 511 33.0 98 33.0
Male 1021 67.0 202 67.0
Ethnicity
African American 897 58.6 149 49.7
Hispanic 41 2.7 12 4.0
White 568 37.1 131 43.7
Eligible for free or reduced lunch 1225 80.0 224 75.0
IEP Labels
Speech/Language 137 7.9 17 5.2
Learning Disabled 1011 58.6 224 67.9
Educable Mentally Disabled 312 18.1 46 13.9
Other Health Impaired 123 7.1 23 7.0
Disability Category
1 (Learning Disability) 944 61.6 207 69.0
2 (All others) 588 38.4 93 31.0
23
Methods Data Analysis

• Measurement Invariance
• CFA/ SEM
• Step 1 one-factor model was established for each
  group individually
• Step 2 one-factor model was established for each
  group simultaneously
• Step 3 three levels of invariance were tested
  using ?? 2
• congeneric (no equality constraints),
• tau-equivalent (set factor loadings equal),
• parallel (set equal factor loadings and error
  variances).
• DOK1 was selected as the reference variable and
  the path (?1) from DOK1 to the ELA factor was set
  to 1.
• Jöreskog, 1971
• Goodness-of-Fit Indices and Criterion
• ?2 chi-squared, insignificant indicates
  measurement invariance
• CFIcomparative fit index ( .95 or above)
• SRMRstandardized root mean square residual (.08
  or below)
• RMSEA root mean square error of approximation
  (.06 or below)
• Hu Bentler, 1999

24
Methods Measurement Invariance

• One-Factor Model for Each Administration
  Mode 
• Note. ? independent variable error variance ?
  factor loadings ?latent disturbance variance

25
Methods Data Analysis

• Differences in Student Performance
• MANCOVA with prior ELA ability as a covariate
• Prior ELA ability total score on 2006 ELA PACT
• Dependent variables
• DOK1, DOK2, DOK3, DOK4, CR, ER
• Xijk µ0 (Effect of COVAR) (Main Effect
  of MODE) (Main Effect of TYPE) (Interaction
  Effect MODETYPE) Residual
• All analyses conducted separately for each grade
  because the tests are not on the same scale.

26
Fit of One-factor Model Grade 6
Goodness-of-fit Indices of the One Factor Model
for Grade 6
Mode df 2 p-value CFI SRMR RMSEA
OS 9 34.41 lt .01 .99 .019 .041
CD-ROM 9 14.36 0.11 .99 .029 .047
Unstandardized (Standardized) Parameter Estimates
of the One-Factor Model Grade 6
Factor Loadings Factor Loadings Error Variances Error Variances
OS CD-ROM OS CD-ROM
DOK1 1.00 (.77) 1.00 (.84) 2.75 (.41) 2.22 (.30)
DOK2 .85 (.71) .74 (.71) 2.89 (.50) 2.85 (.50)
DOK3 .72 (.69) .51 (.61) 2.24 (.52) 2.32 (.63)
DOK4 .53 (.59) .52 (.67) 2.08 (.65) 1.71 (.55)
CR .33 (.60) .27 (.58) .78 (.64) .74 (.66)
ER 1.50 (.58) 1.48 (.59) 17.38 (.66) 22.07 (.66)
27
Fit of One-factor Model Grade 7
Goodness-of-fit Indices of the One Factor Model
for Grade 7
Mode df 2 p-value CFI SRMR RMSEA
OS 9 32.60 lt .01 .99 .018 .039
CD-ROM 9 12.75 .17 1.00 .026 .037
Unstandardized (Standardized) Parameter
Estimates of the One-Factor Model Grade 7
Factor Loadings Factor Loadings Error Variances Error Variances
OS CD-ROM OS CD-ROM
DOK1 1.00 (.68) 1.00 (.59) 2.48 (.53) 2.79(.66)
DOK2 .79 (.62) .94 (.60) 2.23 (.62) 2.25 (.64)
DOK3 .70 (.57) .92 (.60) 2.17 (.67) 2.18 (.64)
DOK4 .79 (.59) 1.23 (.69) 2.52 (.65) 2.38 (.52)
CR .77 (.75) .96 (.75) 1.00 (.44) 1.05 (.44)
ER 2.17 (.59) 3.51 (.70) 18.78 (.65) 19.03 (.51)
28
Fit of One-factor Model Grade 8

Goodness-of-fit Indices of the One Factor Model
for Grade 8
Mode df 2 p-value CFI SRMR RMSEA
OS 9 18.94 .026 1.0 .014 .027
CD-ROM 9 15.53 .07 .99 .031 .051
Unstandardized (Standardized) Parameter Estimates
of the One-Factor Model Grade 8
Factor Loadings Factor Loadings Error Variances Error Variances
OS CD-ROM OS CD-ROM
DOK1 1.00 (.60) 1.00 (.60) 1.00 (.62) 2.51 (.638) 2.82 (.64)
DOK2 1.15 (.62) 1.15 (.62) .89 (.59) 2.97 (.705) 2.65 (.61)
DOK3 1.25 (.64) 1.25 (.64) .97 (.59) 3.28 (.791) 3.10 (.60)
DOK4 1.20 (.68) 1.20 (.68) 1.05 (.69) 2.31 (.808) 2.13 (.53)
CR .96 (.72) .96 (.72) .77 (.64) 1.19 (.799) 1.53 (.48)
ER 2.81 (.60) 2.81 (.60) 2.74 (.64) 20.26 (.812) 18.65 (.64)
29
Summary of Measurement Invariance Findings
Grade 6 Grade 6 Grade 6 Grade 6 Grade 6 Grade 6 Grade 6 Grade 6 Grade 6
Model 2 df ? 2 ? 2 df p-value CFI SRMR RMSEA
Congeneric 52.18 19 - - - .99 .06 .04
Tau-equivalent 62.2 24 10.02 5 .07 .99 .07 .04
Parallel 72.26 30 10.06 6 .12 .99 .09 .04
Grade 7 Grade 7 Grade 7 Grade 7 Grade 7 Grade 7 Grade 7 Grade 7 Grade 7
Model 2 df ? 2 ? 2 df p-value CFI SRMR RMSEA
Congeneric 49.62 19 - - - .99 .06 .04
Tau-equivalent 69.27 24 19.65 5 .001 .99 .09 .04
Parallel 71.48 30 2.21 6 .89 .99 .1 .04
Grade 8 Grade 8 Grade 8 Grade 8 Grade 8 Grade 8 Grade 8 Grade 8 Grade 8
Model 2 df ? 2 ? 2 df p-value CFI SRMR RMSEA
Congeneric 35.52 19 - - - 1.0 .04 .03
Tau-equivalent 42.45 24 6.93 5 .22 1.0 .06 .03
Parallel 53.17 30 10.72 6 .09 1.0 .07 .03
Note plt05
30
Summary of Means by Delivery Mode Grade 6
(question 2a)
Model Mean SD
2006 ELA PACT OS 25.41 7.41
CD 26.59 7.66

DOK1 OS 6.70 2.60
CD 6.63 2.73

DOK2 OS 5.73 2.41
CD 5.56 2.39

DOK3 OS 5.07 2.08
CD 5.09 1.92

DOK4 OS 3.87 1.79
CD 3.96 1.77

CR OS .87 1.11
CD .85 1.06

ER OS 14.62 5.14
CD 14.83 5.80
31
Summary of Means by Delivery Mode Grade 7
(question 2a)
Model Mean SD
2006 ELA PACT OS 34.10 11.60
CD 35.11 12.66

DOK1 OS 5.04 2.15
CD 5.09 2.06

DOK2 OS 3.91 1.89
CD 3.85 1.88

DOK3 OS 4.25 1.80
CD 4.30 1.85

DOK4 OS 4.57 1.97
CD 4.74 2.14

CR OS 1.78 1.51
CD 1.74 1.55

ER OS 15.39 5.39
CD 15.48 6.08
32
Summary of Means by Delivery Mode Grade 8
(question 2a)
Model Mean SD
2006 ELA PACT OS 33.73 10.93
CD 33.88 11.03

DOK1 OS 4.53 1.98
CD 4.58 2.14

DOK2 OS 5.50 2.20
CD 5.44 2.01

DOK3 OS 5.41 2.35
CD 5.45 2.18

DOK4 OS 4.73 2.08
CD 4.67 2.01

CR OS 1.50 1.58
CD 1.54 1.60

ER OS 16.20 5.61
CD 16.45 5.64
33
Summary of Means by Delivery Mode and Disability
Grade 6 (question 2b)

• Ranged from 0.01 to 2.9.
• These small differences are consistent with the
  non-significant interaction effect.

34
Summary of Means by Delivery Mode and Disability
Grade 7 (question 2b)

• Ranged from 0.02 to 0.42.
• These small differences are consistent with the
  non-significant interaction effect.

35
Summary of Means by Delivery Mode and Disability
Grade 8 (question 2b)

• Ranged from 0.04 to 2.61.
• These small differences are consistent with the
  non-significant interaction effect.

36
Results Differences in Student Performance
Grade 6 Grade 7 Grade 8
Main Effect (MODE) Wilks lamda .99 .99 .99
Main Effect (MODE) F 1.3 0.94 .25
Main Effect (MODE) df 6, 1930 6, 2053 6, 1822
Main Effect (MODE) p-value .26 .46 .96
Interaction (MODETYPE) Wilks lamda .99 .99 .99
Interaction (MODETYPE) F .67 1.25 .22
Interaction (MODETYPE) df 6, 1930 6, 2053 6, 1822
Interaction (MODETYPE) p-value .67 .28 .97
37
Students Scoring At Each Proficiency Level
Grade 6 Grade 6 Grade 7 Grade 7 Grade 8 Grade 8
Frequency Percent Frequency Percent Frequency Percent
Below Basic 1620 83 1833 88.9 1651 90.1
Basic 301 15 219 10.6 175 9.6
Proficient 19 1 11 0.5 6 0.3
Advanced - - - - - -
38
Limitations

• Implementation of the CD-ROM format
• Opportunity to practice
• Broad disability categories
• Age and content area
• Accommodation decision-making
• Other accommodations besides read-aloud?

39
Recommendations for Future Study

• Further investigating read-aloud accommodation on
  specific disabilities
• Differences in students who received OS vs.
  CD-ROM
• Looking at DOK differently
• Item difficulty
• Analyzing complexity of reading passage
• Investigating how the CD-ROM was used
• Did students repeat questions and passages?