Title: Valid, Reliable
1Valid, Reliable Efficient
- A Psychometric Evaluation of
- Flash Word Recognition and
- NSSI Passage Reading Measures
2Kathleen J. BrownMatthew K. FieldsUniversity of
UtahR. Darrell Morris Appalachian State
University
3Impetus for Current Study
- Need for valid, reliable, efficient instruments
to determine instructional reading level - Flaws with current instruments
- DRA no rate time consuming
- DIBELS screen only
- IRIs psychometric evaluation often weak or
missing
4Impetus for Current Study
- Growing use of Flash selected graded passages
a.k.a. NSSI (Virginia/ASU effect) - Initial psychometric evaluations positive
- (Frye, 2004 Frye Trathen, 2004 Frye Trathen,
Olson, Schlagal, 2002 Palmer, Trathen, Olson,
Schlagal, 2002)
5Theoretical Framework
(Anastasi, 1988 APA, 1985)
6Methods
- 4 schools
- 2 Title 1 1 public, 1 parochial
- 2 non-Title 1 both public mixed SES
- 192 students in G2-G5 in March, 2006
- Rank ordered DIBELS or QRI, then sampled 12
students per grade 4 high, 4 average, 4 poor to
achieve a representative distribution for testing
7Methods
- 135 minutes of assessment in 3 sessions
- Presentation order counterbalanced
- Flash item selection counterbalanced
- 9 on data team 4 hours protocol training
- Manual flash interrater differences n.s.
8Alternate Form Reliability
- measure of temporal stability for scores
- measure of consistency of response for scores
9Alternate Form Reliability
- To what extent are NSSI A passage scores
equivalent to NSSI B passage scores? - To what extent are computer Flash scores
equivalent to manual Flash scores?
10(No Transcript)
11(No Transcript)
12(No Transcript)
13Results Alternate Form Reliability
p lt .01
14Content Validity
- provides evidence that items on test represent a
specific domain - provides evidence that the format and response
properties of the test represent the domain
15Content Validity
- To what extent do the NSSI passages
reflect/measure expected grade level benchmarks? - Maybe look at separate means for accuracy, rate,
comp report those to show - To what extent does the Flash measure reading
instructional level?
16NSSI Reading Level Criteria
17 Performance Levelfor NSSI by Grade
18Performance Levelfor Flash by Format Criterion
19Concurrent Validity
- To what extent are Flash scores and NSSI scores
consistent with scores achieved on a flagship
standardized reading measure (i.e., the GORT)?
20(No Transcript)
21Results Concurrent Validity
p lt .01
22Average Performance Levelfor NSSI GORT by
Grade Level
23Conclusions For G2-G5
- NSSI A and NSSI B seem to have high validity for
identifying students instructional reading
levels - NSSI A and NSSI B can be considered equivalent
forms
24Conclusions For G2-G5
- Manual Flash and Computer Flash seem to have high
validity for identifying students instructional
reading levels when the criterion is set at 85 - Manual Flash and Computer Flash seem to be
equivalent forms
25Conclusions For G2-G5
- The GORT does not seem to have high validity for
identifying students instructional levelsat any
grade level. - The GORT over-predicts instructional levelby
approx. 2 years. - Note most GORT comp questions are passage
independent (Keenan Betjemann, 2006)