Academic Improvement Monitoring System

1

• Academic Improvement Monitoring System

2
AIMSweb Booster Session

• Series 2 Schools Initial SWAT
• August 27, 2008
• Presenters L. Williams K. Todd

3

• AIMSweb Booster Training
• Outcomes
• Participants will receive a review of
• General Outcome Measures CBMs
• Admin/Scoring of R-CBM MAZE
• Admin/Scoring of LNF LSF
• Ideas for Benchmark Implementation
• Year One Expectations

4
OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A
GENERAL OUTCOME MEASURE

• Mark R. Shinn, Ph.D.
• Michelle M. Shinn, Ph.D.

5
Big ideas of Benchmark Assessment

• Its about using General Outcome Measures (GOMs)
  for formative assessment/evaluation to
• Inform teaching AND
• ensure accountability.
• Its different from, but related to, summative
  high-stakes testing/evaluation, which
• Doesnt inform teaching.
• Mostly used for accountability/motivation.

6
Common Characteristics of GOMs

• The same kind of evaluation technology as other
  professions
• Powerful measures that are
• Simple
• Accurate
• Efficient indicators of performance that guide
  and inform a variety of decisions
• Generalizable thermometer that allows for
  reliable, valid, cross comparisons of data

7
Skill Areas Currently Assessable via AIMSweb

• Early LiteracyK-1 benchmark, Progress Monitor
  (PM) any age
• Letter Naming Fluency
• Letter sound fluency
• Phonemic Segmentation Fluency
• Nonsense Word Fluency
• Early Numeracy (K-1 benchmark, PM any age)
• Oral Counting
• Number identification
• Quantity discrimination
• Missing number
• Oral Reading (K-8, PM any age)
• MAZE (Reading comprehension) (1-8, PM any age)
• Math Computation (1-6, PM any age)
• Math Facts (PM any age)
• Spelling (1-8, PM any age)
• Written Expression (1-8, PM any age)

8
Big Ideas of Benchmark (Tier 1) Assessment

• Benchmarking allows us to add systematic
  Formative Evaluation to current practice.
• For Teachers (and Students)
• Early Identification of At Risk Students
• Instructional Planning
• Progress Monitoring
• For Parents
• Opportunities for Communication/Involvement
• Accountability
• For Administrators
• Resource Allocation/Planning and Support
• Accountability

9
2006
School Calendar Year (2006-2007)
Benchmarking (Tier 1)
2007
2-weeks duringSeptember 1 to October 15 January
1 to February 1 May 1 to June 1
10
Administration and Scoring of READING-CURRICULUM
-BASED MEASUREMENT (R-CBM) for Use in General
Outcome Measurement

• Mark R. Shinn, Ph.D.
• Michelle M. Shinn, Ph.D.

11
Things you Need Before Testing

• 1. Standard Reading Assessment Passage Student
  Copy
• No numbers
• Between 250-300 words (exception 1st grade)
• An informative first sentence
• Same font style and size
• Text without pictures
• Obtain from your LAM

12
Things you Need Before Testing

• 2. Standard Reading Assessment Passage Examiner
  Copy
• Pre-numbered so they can be scored quickly and
  immediately.
• Obtain from your LAM.

13
4. Additional Assessment Aids Needed Before
Testing
A list of students to be assessed Stop Watch
(requireddigital preferred) Clipboard Pencil
Transparencies or paper copies of examiner
passages Dry Marker or Pencil Wipe Cloth (for
transparencies only)
14
Setting up Assessment Environment

• Assessment environments are flexible and could
  include
• A set-aside place in the classroom
• Reading station in the hall way
• Reading stations in the media center, cafeteria,
  gym, or empty classrooms

15
Things You Need to do While Testing

• Follow the standardized directions
• R-CBM is a standardized test
• Administer the assessment with consistency
• Remember its about testing, not teaching
• Dont teach or correct
• Dont practice reading the passages
• Remember best, not fastest reading
• Sit across from, not beside student

16
R-CBM Standard Directions for 1 Minute
Administration

• Place the unnumbered copy in front of the
  student.
• Place the numbered copy in front of you, but
  shielded so the student cannot see what you
  record.
• Say
• When I say Begin, start reading aloud at the
  top of this page. Read across the page
  (DEMONSTRATE BY POINTING). Try to read each word.
  If you come to a word you dont know, I will
  tell it to you. Be sure to do your best reading.
  Are there any questions? (PAUSE)
• Say Begin and start your stopwatch when the
  student says the first word. If the student
  fails to say the first word of the passage after
  3 seconds, tell them the word, mark it as
  incorrect, then start your stopwatch.
• Follow along on your copy. Put a slash ( / )
  through words read incorrectly.
• At the end of 1 minute, place a bracket ( )
  after the last word and say, Stop.
• Score and summarize by writing WRC/Errors

17
Familiar Shortened Directions
When students are assessed frequently and know
the directions. Say When I say Begin, start
reading aloud at the top of this page.
18
Items to Remember
Emphasize Words Read Correctly (WRC). Get an
accurate count. 3-Second Rule. No Other
Corrections. Discontinue Rule. Be Polite. Best,
not fastest. Interruptions.
19
Things to do After Testing
Score immediately! Determine WRC. Put a slash (/)
through incorrect words. If doing multiple
samples, organize your impressions of qualitative
features.
20
What is a Word Read Correctly?
Correctly pronounced words within
context. Self-corrected incorrect words within 3
seconds.
21
What is an Error?
Mispronunciation of the word Substitutions
Omissions 3-Second pauses or struggles
(examiner provides correct word)
22
What is not Incorrect? (Neither a WRC or an Error)
Repetitions Dialect differences Insertions
(consider them qualitative errors)
23
Example
Juan finished reading after 1 minute at the 145th
word, so he read 145 words total. Juan also made
3 errors. Therefore his WRC was 142 with 3
errors. Reported as 142/3.
24

• Benchmark Data Obtain MEDIAN score for
  students 3 passages
• 67 / 2 85 / 8 74
  / 9

1 min. 1 min.
1 min.
Why use Median vs. Average? Averages are
susceptible to outliers when dealing with small
number sets.Median Score is a statistically more
reliable number than average for R-CBM.
25
The Data Obtain MEDIAN score for 3
passages 67 / 2 85 / 8
74 / 9
1 min. 1 min.
1 min.
1. Throw out the HIGH and LOW scores for
Words Read Correct (WRC)
26

• The Data Obtain MEDIAN score for 3 passages
  67 / 2 85 / 8 74 /
  9

1 min. 1 min.
1 min.
2. Throw out the HIGH and LOW scores for the
Errors. Remaining scores MEDIAN. 3.
Report this score in your AIMSweb account.

74/8
27
Lets Practice

• Please refer to
• Practice Exercise 7 in Handouts

28
Determining Inter-Rater Agreement
Example Dave 2 examiners observed Daves
reading 1 scored Dave as 100 WRC 1 scored Dave
as 98 WRC They agreed that Dave read 98 of
the words correct. They disagreed on 2 words
correct. Inter-rater Agreement
FormulaAgreements/(Agreements Disagreements)
x 100IRA (98)/ (98 2) 98/100 .98 .98 x
100 98 Inter-rater Agreement for Dave is
98. (Goal is 95 or better.)
See R-CBM Workbook Page 14
29
Administration and Scoring of READING-MAZE
(R-MAZE) for Use in General Outcome Measurement

• Power Point Authored by
• Jillyan Kennedy
• Based on Administration and Scoring of Reading
  R-MAZE for Use
• with AIMSweb Training Workbook
• By
• Michelle M. Shinn, Ph.D.
• Mark R. Shinn, Ph.D

30
Curriculum Based Measurement Reading R-MAZE
CBM R-MAZE is designed to provide educators a
more complete picture of students reading
skills, especially when comprehension problems
are suspected.
31
Curriculum Based Measurement Reading R-MAZE

• R-MAZE is a multiple-choice cloze task that
  students complete while reading silently.
• The students are presented with 150-400 word
  passages.
• The first sentence is left intact.
• After the first sentence, every 7th word is
  replaced with three word choices inside a
  parenthesis.
• The three choices consist of

• Near Distracter
• Exact Match
• Far Distracter

32
Sample Grade 4 R-MAZE Passage
33
Items Students Need Before Testing

• What the students need for testing
• CBM R-MAZE practice test
• Appropriate CBM R-MAZE passages
• Pencils

34
Setting up Assessment Environment
Assessment environments are flexible and could
include

• The classroom if assessing the entire class.
• A cluster of desks or small tables in the
  classroom for small group assessment.
• Individual desks or stations for individual
  assessment.

35
Things You Need to do While Testing
Follow the standardized directions

• Attach a cover sheet that includes the practice
  test so that students do not begin the test
  right away.
• Do a simple practice test with younger
  students.
• Monitor to ensure students are circling answers
  instead of writing them.
• Be prepared to Prorate for students who may
  finish early.
• Try to avoid answering student questions.
• Adhere to the end of timing.

36
CBM R-MAZE Standard Directions

• Pass R-MAZE tasks out to students. Have students
  write their names on the cover sheet, so they do
  not start early. Make sure they do not turn the
  page until you tell them to.
• Say this to the student (s)
• When I say Begin I want you to silently read a
  story. You will have 3 minutes to read the story
  and complete the task. Listen carefully to the
  directions. Some of the words in the story are
  replaced with a group of 3 words. Your job is to
  circle the 1 word that makes the most sense in
  the story. Only 1 word is correct.
• Decide if a practice test is needed. Say . . .
• Lets practice one together. Look at your first
  page. Read the first sentence silently while I
  read it out loud The dog, apple, broke, ran
  after the cat. The three choices are apple,
  broke, ran. The dog apple after the cat. That
  sentence does not make sense. The dog broke
  after the cat. That sentence does not make
  sense. The dog ran after the cat. That
  sentence does make sense, so circle the word ran.
  (Make sure the students circle the word ran.)

37
CBM R-MAZE Standard Directions (Continued)
Lets go to the next sentence. Read it
silently while I read it out loud. The
cat ran fast, green, for up the hill. The three
choices are fast, green, for up the
hill. Which word is the correct word
for the sentence? (The students answer
fast) Yes, The cat ran fast up the
hill is correct, so circle the correct
word fast. (Make sure students circle fast)
Silently read the next sentence and raise
your hand when you think you know the
answer. (Make sure students know the correct
word. Read the sentence with the correct
answer) Thats right. The dog barked
at the cat is correct. Now what do
you do when you choose the correct
word? (Students answer Circle it.
Make sure the students understand the task)
Thats correct, you circle it. I think youre
ready to work on a story on your own.
38
CBM R-MAZE Standard Directions (Continued)

• Start the testing by saying . . .
• When I say Begin turn to the first story and
  start reading silently. When you come to a group
  of three words, circle the 1 word that makes the
  most sense. Work as quickly as you can without
  making mistakes. If you finish a/ the page/first
  side, turn the page and keep working until I say
  Stop or you are all done. Do you have any
  questions?
• Then say, Begin. Start your stopwatch.
• Monitor students to make sure they understand
  that they are to circle only 1 word.
• If a student finished before the time limit,
  collect the students R-MAZE task and record the
  time on the students test booklet.
• At the end of 3 minutes say Stop. Put your
  pencils down. Please close your booklet.
• Collect the R-MAZE tasks.

39
CBM R-MAZE Familiar Directions

• After the students have put their names on the
  cover sheer, start the testing by saying . . .
• When I say Begin turn to the first story and
  start reading silently. When you come to a group
  of three words, circle the 1 word that makes the
  most sense. Work as quickly as you can without
  making mistakes. If you finish a/ the page/first
  side, turn the page and keep working until I say
  Stop or you are all done. Do you have any
  questions?
• Then say, Begin. Start your stopwatch.
• Monitor students to make sure they understand
  that they are to circle only 1 word.
• If a student finished before the time limit,
  collect the students R-MAZE task and record the
  time on the students test booklet.
• At the end of 3 minutes say Stop. Put your
  pencils down. Please close your booklet.
• Collect the R-MAZE tasks.

40
Things to Do After Testing

• Score immediately to ensure accurate results!
• Determine the number of words (items) correct.
• Use the answer key and put a slash (/) through
  incorrect words.

41
CBM R-MAZE Scoring
What is correct? The students circles the word
that matches the correct word on the scoring
template.

• What is incorrect?
• An answer is considered an error if the student
• Circles an incorrect word
• Omits word selections other than those the
  student
• was unable to complete before the 3 minutes
• expired

42
Making Scoring Efficient

• Count the total number of items up to the last
  circled word.
• Compare the student answers to the correct
  answers on the scoring template. Mark a slash
  / through incorrect responses.
• Subtract the number of incorrect answers from the
  total number of items attempted.
• Record the total number of correct answers on the
  cover sheet followed by the total number of
  errors (e.g., 35/2).

43
Lets Practice

• Please locate your MAZE Practice Exercises page
  in your handout.
• Please score the example.
• Be prepared to share your answer.

44
Administration and Scoring ofEarly Literacy
Measures for Use with AIMSweb
Power Point Created by Jillyan Kennedy Based on
Administration and Scoring of Early Literacy
Measures for Use with AIMSweb Training
Workbook Michelle M. Shinn, Ph.D. Mark R. Shinn,
Ph.D.
45
AIMSweb Early Literacy Measures (Based, in part,
on DIBELS)

• The National Reading Panel (Panel 2000)
  identified the following to be the best
  predictors of how well children will learn to
  read in Kindergarten and Grade 1
• Phonemic Awareness
• Elements of Phonics including letter names,
  letter sounds, and the ability to read nonsense
  words.

46
AIMSweb Early Literacy Measures
47
Recommended AIMSweb Early Literacy Benchmark
Assessment Schedule
48
Administration and Scoring Letter Naming Fluency
49
Things You Need Before Testing Letter Naming
Fluency

• Specific Materials Arranged
• Student copy of Letter Naming Fluency (not
  numbered)
• Examiner copy of Letter Naming Fluency (numbered
  for easy scoring)
• Clipboard to provide a hard surface for recording
  student answers
• Stopwatch

50
Things You Need While Testing Letter Naming
Fluency
Standard Directions for 1- Minute Administration

• Place the unnumbered copy in front of the
  student.
• Place the numbered copy in front of you, but
  shielded so the student cannot see what you
  record.
• Say Here are some letters point to the student
  copy. Begin here, point to the first letter
  and tell me the names of as many letters as you
  can. If you come to a letter you dont know,
  Ill tell it to you. Are there any questions?
  Put your finger under the first letter. Ready,
  begin.
• Start you stopwatch. If the student fails to say
  the first letter name after 3 seconds, tell the
  student the letter name and mark it as incorrect.
  Point to the next letter to indicate for the
  child to move on.
• If the student provides the letter sound rather
  than the letter name say, Remember to tell me
  the letter name, not the sound it makes. This
  prompt may be provided once during the
  administration.

51
Things You Need While Testing Letter Naming
Fluency
Standard Directions for 1- Minute Administration
(Continued)

• If the student does not get any correct letter
  names within the first 10 letters (1 row),
  discontinue the task and record a score of 0.
• Follow along on your copy. Put a slash ( / )
  through letter names given incorrectly.
• The maximum time for each letter is 3 seconds.
  If a student does not provide the next letter
  within 3 seconds, tell the student the letter
  name and mark it as incorrect. Point to the next
  letter and say, What letter?
• At the end of 1 minute, place a bracket ( )
  after the last letter and say, Stop.

52
What is a Correct Letter Name?

• A correctly named letter
• Confused Is and Ls as a function of font
• Self Corrections

53
What is an Incorrect Letter Name?

• Substitutions of a different letter for the
  stimulus letter (e.g., P for D)
• Omissions of a letter
• Stops or struggles with a letter for more than
  3 seconds
• Note Skipped Row. If a student skips an entire
  row, draw a line through the row and do not count
  the row in scoring

54
Calculating Reporting Scores Letter Naming
Fluency

• Count the total number of letters the student
  read
• Count the number of errors and subtract
• Report the total number of letters named
  correctly

55
LNF

• Show DVD Practice Example.

56
Administration and Scoring Letter Sound Fluency
57
Things You Need Before Testing Letter Sound
Fluency

• Specific Materials Arranged
• Student copy of Letter Sound Fluency (not
  numbered)
• Examiner copy of Letter Sound Fluency (numbered
  for easy scoring)
• Clipboard to provide a hard surface for recording
  student answers
• Stopwatch

58
Things You Need While Testing Letter Sound Fluency
Standard Directions for 1- Minute Administration

• Place the unnumbered copy in front of the
  student.
• Place the numbered copy in front of you, but
  shielded so the student cannot see what you
  record.
• Say, Begin here, point to the first letter and
  tell me the sounds of as many letters as you can.
  If you come to a letter you dont know, Ill
  tell it to you. Are there any questions? Put
  your finger under the first letter. Ready,
  begin.
• Start your stopwatch. If the student fails to
  say the first letter sound after 3 seconds, tell
  the student the letter sound and mark it as
  incorrect. Point to the next letter and say,
  What sound?
• If the student provides the letter name rather
  than the letter sound say, Remember to tell me
  the sound the letter makes, not its name. This
  prompt may be provided once during the
  administration.

59
Things You Need While Testing Letter Sound Fluency
Standard Directions for 1- Minute Administration
(Continued)

• If the student does not get any correct letter
  sounds within the first 10 letters (1 row),
  discontinue the task and record a score of 0.
• Follow along on your copy. Put a slash ( / )
  through letter sounds given incorrectly. The
  maximum time for each letter is 3 seconds.
• The maximum time for each letter is 3 seconds.
  If a student does not provide the next sound
  within 3 seconds, tell the student the letter
  sound and mark it as incorrect. Point to the
  next letter and say, What sound?.
• At the end of 1 minute, place a bracket ( )
  after the last letter and say, Stop.

60
What is a Correct Letter Sound?

• Students provide the most COMMON sound of the
  letter. (A pronunciation guide for the most
  common sounds is in the Appendix of the
  AIMSweb Training Workbook)
• Confused Is and Ls as a function of font
• Self Corrections

61
What is an Incorrect Letter Sound?

• Substitutions of a different letter sound for
  the stimulus letter (e.g., /puh/ for /D/)
• Omissions of a letter sound
• Stops or struggles with a letter sound for more
  than 3 seconds
• Note Skipped Row. If a student skips an entire
  row, draw a line through the row and do not
  count the row in scoring

62
LSF

• Show DVD Practice Example.

63
Expectations for Series 2 Schools

• Complete Universal Screenings for grades K-2
• Develop confidence and competence administering
  and scoring the CBMs
• Complete AIRS for SWAT members

64
Expectations for Series 2 Schools (cont)

• Complete the RtI Self-Assessment

65
Reaching Beyond Expectations

• Conduct Grade Level reviews of the benchmark data
• Identify reading interventions available in/to
  your school
• Establish Core Team to set RtI goals for your
  school

66
The End

• This concludes the AIMSweb Booster session for
  Series 2 Initial SWATs
• QUESTIONS????
• Remember to contact us for support!
• GOOD LUCK TO YOU ALL!!