CurriculumBased Measures in Reading: Age and Skilllevel Comparisons

1
Curriculum-Based Measures in
Reading Age and Skill-level Comparisons
RESEARCH INSTITUTE ON PROGRESS MONITORING Institu
te on Community Integration / Special Education
Programs University of Minnesota Funded through
the U.S. Department of Education, CFDA 84.324H

Curriculum-Based Measurement Mirkin
Conference May 5, 2006, Minneapolis, MN
2
Research Team

• Teri Wallace
• Jeff Long
• Renata Ticha
• Miya Miura Wayman
• Hilda Ives Wiley
• Sally Du
• Minneapolis School District
• St. Croix River Education District

3
Overview of Presentation

• What is CBM?
• Previous research in reading
• Elementary-school
• Secondary-school
• Current research Across age and skill levels
• Background
• Research questions
• Participants
• CBM Measures
• Criterion variables
• Procedures
• Results
• Scoring procedures
• Reliability
• Concurrent validity
• Growth over time
• Relation of growth to criterion variables
• Summary
• Implications

4
WHAT IS CURRICULUM-BASED MEASUREMENT?
5
Curriculum-Based Measurement

• Simple procedure for repeated measurement of
  student growth toward long-range instructional
  goals (Deno, 1985)

Simple
Repeated measurement
Long-range goals
6
Picture of Progress Monitoring Using CBM
Measures
1. Determine current level of performance
2. Set individual annual goal -- tied
to
state standards.
4. Evaluate and modify instruction on basis of
rate of progress towards goal.
3. Monitor individual progress towards goal.
7
Case Study
8
Case Study Jennifer

• Jennifer is a 16 year old tenth grader who
  receives special education support for reading.
  Jennifer's special education teacher and her
  English teacher are concerned about her
  performance in reading. Jennifer's performance in
  reading is markedly below her peers, and she is
  having trouble passing the state graduation
  standards test in reading. Jennifer's teachers
  decides to monitor Jennifer's progress in
  reading. Her special education teacher collects
  baseline data on Jennifer. Jennifer's baseline
  scores are 19, 17, and 32 correct maze choices in
  3 minutes.

9
Sample Maze for Jennifer

• Kids Try Out the World of Work in a City Thats
  Just Their Size
• The youthful mayor of the kid-sized city in
  Maplewood clearly has a good grasp of politics.
  My approach is that I try it / he / to be
  friendly, so everyone will like his / a / me,
  said 11-year old Justin Smith. End / He / Age
  was mayor for a day in Exchange City. This / Able
  / From is Junior Achievements new hands-on
  another / responded / learning center that helps
  students understand what its / here / who like
  to work in the real spent / world / gown. Junior
  Achievement is an international nonprofit
  recognizing / complicated / organization. It is
  dedicated to educating and inspiring / voyagers /
  occasion youth about business and economics.
• Exchange City several / opened / schools last
  week in a former bank speaking / building /
  primary on White Bear Avenue. Eighty
  sixth-graders with / here / from Pullman
  Elementary in St. Paul Park farm / were / join
  the inaugural population. They spent Monday
  working / father / during in shops and earning a
  paycheck.

Article adapted from the Minneapolis Star Tribune
10
Sample Graph for Jennifer
11
PREVIOUS RESEARCH IN READING
12
Elementary-school Level
13
Sample Measures Elementary
Reading aloud Mike watched with envy as
Eddie and Joe climbed onto their bikes. Mike knew
how Eddie and Joe had gotten their bikes. Eddie
had begged his parents until he got a bright
yellow racing bike for his birthday. Joe had
pleaded until a blue mountain bike had showed up
under the Christmas tree. Mike tried begging and
pleading with his parents, but he still did not
have a bike.
Scoring words read correctly Time 1
minute
Maze selection Mike watched with envy as
Eddie and Joe climbed onto their bikes. Mike knew
how Eddie and Joe up / had / to gotten their
bikes. Eddie had begged sat / his / rid parents
until he got a bright yellow / winter / banner
racing bike for his birthday. Joe for / kite /
had pleaded until a blue mountain fort / bike /
rate had showed up under the Christmas tree. Mike
cold / tried / drive begging and pleading with
his parents, get / the / but he still did not
have a bike / tale / were.
Scoring correct choices / correct minus
incorrect choices Time 1 or 2 minutes
14
Reading aloud Validity
(Fuchs, Fuchs, Maxwell, 1988)
15
Reading aloud Growth
Deno, 1985
16
Maze selection Validity
Jenkins Jewell, 1993
17
Maze selection Growth
Jenkins Jewell, 1993
18
Secondary-school Level
19
Sample Measures Secondary

• Reading aloud
• Larry Fryklund has marked many milestones
  in his 86 years of life. On Wednesday night, he
  marked another one he graduated from high
  school. Dressed in his black cap and gown, he
  shook hands with commencement speaker, ex-Viking,
  and state Supreme Court Justice Alan Page. Then
  he took his diploma from Principal Linda Nelson.

Scoring words read, words read correctly
Time 1, 2, and 3 minutes
Maze selection Larry Fryklund has marked
many milestones in his 86 years of life. On
Wednesday night, he marked another was / eat /
one he graduated from high school. Dressed be /
in / or his black cap and gown, he shook / carts
/ south hands with commencement speaker,
ex-Viking, will / and / buy state Supreme Court
Justice Alan Page. Hole / Said / Then he took
his diploma from Principal Linda Nelson.
Scoring correct and correct minus incorrect
choices Time 2, 3, and 4 minutes
20
Alternate-form Reliability
Espin, Wallace, Lembke,Campbell, Long, 2003
21
Criterion-related Validity
22
Growth Reading aloud
Words read correctly 3 minutes
23
Growth Maze selection
Correct maze choices 3 minutes
24
Validity Follow-up study
Ticha, Miura, Nanayakkara, Espin, 2004
25
(No Transcript)
26
(No Transcript)
27
Validity of Growth Rates

• Growth rates on maze selection significantly
  related to
• Performance on MBSTs (both studies)
• Pre/post change on WJIII
• Level of reading performance as identified by
  teachers

28
(No Transcript)
29
CURRENT RESEARCH IN READING
30
Across Elementary and Secondary
31
Typical Monitoring Within Year
1. Determine current level of performance
2. Set individual annual goal -- tied
to
state standards.
4. Evaluate and modify instruction on basis of
rate of progress towards goal.
3. Monitor individual progress towards goal.
32
Growth Charts Height Weight (2-20 yrs)
OK135S067
33
Seamless and Flexible System of Progress
Monitoring
Typical growth rate
Mikes growth rate
Kellys growth rate
Sues growth rate
K 1 2 3 4 5
6 7 8 9 10
11 12
34
Research Questions

• What are the validity and reliability of reading
  progress measures across age and skill levels?
• Are validity and reliability influenced by
• Scoring procedure
• Type of measure
• Passage difficulty
• Duration?
• Do the measures reflect change in performance
  from Fall to Winter to Spring?

35
Participants (n505)

• Age levels
• Grade 3 111
• Grade 5 130
• Grade 8 90
• Grade 10 178
• Skill levels Range of skill levels students
  taking state graduation tests

36
Participants (cont.)

• Schools 1 Urban and 1 Rural District
• Urban 2 schools, K-8, 9-12
• Rural 4 schools, K-5 (2), 6-8, 9-12
• Demographic information
• Gender Male (47) Female (53)
• Special education SLD, speech, POHI, DHH, EBD,
  Autism, Early childhood (7)
• Ethnicity African-American (17), Asian-American
  (5), Hispanic American (12), White (65),
  Native American (1)
• Language ELL (11)
• Free / reduced lunch Yes (34)

37
Participants (cont.)

• Schools 1 Urban and 1 Rural District
• Urban 2 schools, K-8, 9-12
• Rural 4 schools, K-5 (2), 6-8, 9-12
• Demographic information
• Gender Male (239) Female (270)
• Special education 37 students (SLD, speech,
  POHI, DHH, EBD, Autism, Early childhood)
• Ethnicity African-American (87), Asian-American
  (23), Hispanic American (60), White (328), Native
  American (7)
• Language ELL (58), non-ELL (450)
• Free / reduced lunch Yes (173) No (332)

38
CBM Measures

• Reading aloud
• Score Words read correctly
• Duration 1 minute
• Difficulty Common vs. grade-level passage
• Maze selection
• Score
• Correct vs. correct minus incorrect choices
• 2 vs. 3- In-a-Row Rule
• Duration 1, 2, and 3 minutes
• Difficulty Common vs. grade-level

39
Criterion Variables

• Norm-referenced, standardized achievement test
  NALT / MAP
• Northwest Evaluation Association
• Achievement Level Tests (Paper pencil)
• Measures of Academic Progress (Computer)
• State Reading Test MCAs
• Minnesota Comprehensive Assessments
• State Graduation Test MBSTs
• Minnesota Basic Skills Tests

40
Procedures

• CBM Data collection rounds Fall, Winter, Spring
• CBM Data collected each round
• 12 passages total (6 passages in both reading
  aloud and maze form)
• 6 reading aloud 3 common, 3 grade-level
• 6 maze 3 common, 3 grade-level
• Data collected over two sessions, one week apart
• Everything counterbalanced within round (i.e.,
  Fall, Winter, and Spring data collection
  identical)
• Criterion variables
• Administered by Districts
• Winter MBST Spring NALT / MAP

41
Results

• Scoring procedures
• Reliability
• Age / Skill
• Type of measure
• Passage difficulty
• Duration
• Validity
• Age / Skill
• Type of measure
• Passage difficulty
• Duration
• Growth Fall to Winter to Spring
• Reflect growth?
• Growth valid?

42
Scoring ProceduresCorrect vs. Correct-Incorrect

• No differences in reliability or validity
  Correct vs. correct-incorrect
• True across passage difficulty, duration, and
  grade
• Select Correct

43
Scoring Procedures2- vs. 3- In-a-Row Rule

• No differences in reliability or validity 2 vs.
  3 incorrect
• True across passage difficulty, duration, and
  grade
• Select 3 incorrect

44
Reliability, Validity, Growth Results
Go to Graphs
45
Alternate-form Reliability

• Maze and reading aloud Nearly all above .80
• Reading aloud above .90
• Grade level few differences (see below)
• Common vs. grade-level few differences
• Maze, grade level, 8th grade slightly lower
• Duration maze 2 and 3 minutes tend to produce
  stronger reliabilities than 1 minute
• 1 minute close to or above .80

46
Concurrent Validity NALT / MAP

• Grade-level
• Correlations drop at 8th grade (mostly .40s)
• Similar results with MBSTs (which include entire
  sample)
• Magnitude of correlations different from previous
  research
• Correlations highest at 5th grade (most .70s) and
  somewhat lower at 3rd grade (most .60s)
• Reading aloud maze Similar patterns
• Reading aloud somewhat stronger, especially
  grade-level, 8th grade
• Duration 2 and 3 minute maze stronger than 1
  minute, grades 3 5
• Common and grade-level Patterns similar with
  exception of grade-level reading aloud (not
  replicated with MBSTs)
• Skill level Curvilinear relationships (i.e.,
  curve flattens starts to flatten out at higher
  values of CBM measures)
• Varies with grade level (3rd and 5th) and
  criterion variable
• Effects small
• Could be due to a ceiling in criterion variable?

47
Concurrent validity MCAs

• Patterns similar to NALT/MAP
• Grade level
• Correlations drop at 10th grade, highest at 5th
  grade
• Magnitude similar to NALT/MAP in 3rd (.60s), 5th
  (.70s)
• Maze vs. reading aloud Reading aloud somewhat
  higher at 3rd
• Duration 2 3 minutes stronger than 1 minute
• Common vs. Grade level Similar patterns of
  results similar magnitude of correlations
• Skill level Linear relationships (i.e., no
  interaction between relationship and skill level
  within grade)

48
Growth Fall to Winter to Spring

• Reading aloud
• Significant growth Fall to Winter to Spring, all
  grades
• Common and grade-level passages Amount of F to
  Sp growth decreases as grade level increases (3,5
  similar 8,10 similar)
• Cross over between grades Discuss later
• Maze
• Significant growth Fall to Winter to Spring, all
  grades
• Common and grade-level passages
• Amount of F to Sp growth amazingly similar across
  grades, especially for common passages
• Magnitude of growth increases with duration
• Cross over between grades Discuss later

49
Growth Across grades

• Reading aloud
• Common passage better at detecting growth across
  grades than grade level
• Curvilinear Common passage flattens out grades 8
  to 10
• Maze
• Common passage better at detecting growth across
  grades than grade level
• Not curvilinear Common passage produces
  beautiful growth rates across grades, especially
  at 3 minutes

50
Growth Validity

• CBM Measures Maze, 3 minutes (common and
  grade-level) Reading aloud, 1 minute (common and
  grade-level)
• Growth relates to criterion measures
• Higher score on NALT/MAP and MCAs associated with
  higher CBM intercept and greater increase on CBM
  measures over time
• Even for lower scores on NALT/MAP and MCAs,
  significant relation, but rate of growth less
• Anomaly Reading aloud, grade-level -- No
  significant relation between growth on reading
  aloud, grade-level passage and NALT at any grade
  level

51
Reliability, Validity, Growth Results
Return from Graphs
52
Reading Research Questions

• What are the validity and reliability of reading
  progress measures across age and skill levels?
• Are validity and reliability influenced by
• Scoring procedure
• Type of measure
• Passage difficulty
• Duration?
• Do the measures reflect change in performance
  from Fall to Winter to Spring?

53
Conclusions

• Scoring procedures No difference reliability or
  validity
• Reliability
• Reliabilities acceptable for all measures, all
  grades, all durations
• Maze 2 minutes recommended (somewhat stronger
  than 1 minute at all grade levels)
• Validity
• Age levels Strength of relations between CBM and
  criterion measures drops as students increase in
  age
• Correlations at 8th-grade surprisingly (and
  disappointingly) low
• Skill levels Within grade, correlations remain
  same across skill levels or level off at higher
  ends of CBM score continuum
• Type of measure Reading aloud vs. maze No clear
  patterns of differences (1 minute reading aloud
  stronger than 1 minute maze at each grade level)
• Passage difficulty Common and grade-level
  passages produce similar results (surprising for
  older student)
• Duration Correlations tend to increase with maze
  with time

54
Conclusions (cont.)

• Growth
• All measures reflect change from Fall to Winter
  to Spring
• Amount of change does not differ for common and
  grade-level passage
• Amount of change similar by grade level,
  especially 3 and 5, 8 and 10
• Amount of change similar for common and
  grade-level passage
• Common maze passage Best passage for reflecting
  growth across years
• Changes significantly related to performance on
  NALT / MAP and MCAs
• Reading aloud contradicts previous research??

55
Implications for practice

• Elementary-school Level
• Performance and Progress
• 1 minute reading aloud (words correct)
• 2 minute maze selection (correct choices)
• Secondary-school Level
• Performance
• 1 minute reading aloud (words correct)
• 2 / 3 minute maze selection (correct choices)
• Progress
• 3 minute sample maze (correct choices)
• Elementary to Secondary
• 2 / 3 minute maze, common level passage