Formative Evaluation Data Based Instructional Decision Making

1
Formative Evaluation Data Based Instructional
Decision Making

• Tom Jenkins, Ed.D.
• Educational Consultation Services, LLC
• Wilmington, NC

2
What you can expect today

• CBM
• What is it and how do we do it?
• Instruction on progress monitoring
• Data based instructional decision making
• Examples and hands-on activity
• A complete case study

3
What is CBM?

• Curriculum-based measurement
• Curriculum-based assessment
• Curriculum-based evaluation
• Data collection tools derived directly from the
  curriculum that student is expected to learn

4
CBM

• CBM is believed to reduce the gap between
  assessment and instruction
• Aides teachers in generating superior student
  achievement
• Improved communication
• Higher level of sensitivity
• Enhancement of the database
• Administration time is shorter
• More cost effective

5
CBM

• Mirkin et al. (1982) in a study of 50 teachers
• 90 of the teachers stated that using CBM
  improved IEP objective and goal development,
  student progress monitoring, and instructional
  decision making
• Fuchs and Fuchs (1999) argued that CBM allows for
  IEPs focused on student outcomes instead of a
  laundry list of short-term objectives

6
CBM

• The simplicity of CBM allows for quick and easy
  peer referencing
• Normative data can be collected
• This allows for comparison of a students
  performance to his/her actual peer group
• More representative geographically, culturally,
  ethnically, and has been exposed to similar
  instructional environment

7
CBM

• CBM has been shown to posses high levels of
  reliability
• 42 one-minute CBM type assessments in reading,
  math, and written expression for grade K-5 were
  found to have reliability coefficients between
  .90-.99 with just three one-minute
  administrations (Jenkins, 2002)

8
CBM

• Discriminant Validity
• Several studies have demonstrated the ability of
  CBM to differentiate between students receiving
  special education services, students receiving
  Chapter 1 services, and students not receiving
  any of those services (Deno, Marston, Shinn, and
  Tindal, 1983 Marston and Deno, 1982 Shinn and
  Marston, 1985 and Shinn, Tindal, Spira, and
  Marston, 1987).

9
CBM Procedures

• Remember different modalities
• Given the nature of reading probes they must be
  administered individually
• Math and written expression may be administered
  in a group setting

10
CBM Procedures

• Scoring
• One minute administration time except for written
  expression
• Number ID is scored corrects for minute however,
  math computations are scored digits correct per
  minute
• Written expression is scored according to correct
  sequences

11
Example

• Administration example
• Work in pairs
• Each person gets a chance to administer

12
CBM Procedures

• All reading probes scored corrects per minute

13
CBM Procedures

• Math computations are scored by correct digits
  per minute

14
(No Transcript)
15
CBM Procedures

• Correct Sequences for written expression
• Two words form a sequence, word and punctuation
  form a sequence.
• Most words and punctuation are used twice
• Three minutes to brainstorm, write, and edit

16
(No Transcript)
17
(No Transcript)
18
Progress Monitoring

• Basic Principle 8 of PSM
• Progress monitoring an essential aspect of the
  intervention phase
• Basic Principle 9 of PSM
• Decision making in regards to the effectiveness
  of an intervention is based on analysis of
  progress monitoring data in relation of goal

19
Thoughts and Conclusions

• Progress monitoring and charting are components
  of formative evaluation
• Allows you to determine the effectiveness of an
  intervention during implementation so that it can
  be modified or changed to increase the likelihood
  that intended results will be achieved. (Deno,
  2002)

20
Progress Monitoring

• Essential for four reasons
• There is no guarantee that interventions will be
  successful, thus the intervention must be
  tested to evaluate effectiveness
• Increased emphasis of specific outcomes for
  students, data base must be generated to guide
  intervention decision making
• Pre/post testing has be shown to be unreliable
  (small amount of data) and provides too little
  data to allow for instructional decision making
  progress monitoring allows for evaluation of
  level of performance and rate of learning
• Research has shown that progress monitoring is
  associated with improved educational outcomes

21
Progress Monitoring

• Definition
• Frequent and repeated data collection (dependent
  upon students level/tier) and analysis of
  student performance
• Data is collected during intervention and
  provides basis for analysis of intervention
  effectiveness

22
Progress Monitoring

• Essential components that must be in place for
  successful progress monitoring
• A well-defined behavior
• A measurement strategy
• Identification of students current level of
  performance (baseline)
• Intervention
• Goal
• Graph
• Decision-making plan

23
Well Defined Behavior

• Target behavior, observable, measurable, and
  specific
• Focus on enabling skills
• Skills that are prerequisite skills for more
  complex skills
• Deficiencies in enabling skills often adversely
  affects performance on global assessments

24
Well Defined Behavior

• Enabling skills for reading
• Phonemic awareness
• Alphabetic understanding
• Fluency
• Sight words
• Comprehension

25
Well Defined Behavior

• Enabling skills for math
• Number sense
• Facts
• Computation
• Applications
• Problem solving
• Enabling skills for written expression
• Mechanics
• Expression

26
Well Defined Behavior

• Enabling skills for behavior
• Social skills
• Work completion
• Compliance
• Problem solving skills

27
Measurement Strategy

• Frequent and repeated collection of data
• Time and cost efficient
• Sensitive to changes over short periods of time

28
Current Level of Performance (Baseline)

• Gathered prior to intervention
• Repeatable
• Provides comparison for progress data
• Helps set goal
• Median score why?

29
Intervention

• Match intervention to problem
• Humans tend to employ interventions with which
  they are comfortable instead of intervention that
  the student needs
• Intervention should be developed with the
  expectation that it will be altered in some way
  as a result of the progress monitoring data
• No intervention works all of the time for every
  student

30
Goal

• Standard against which progress can be compared
• Allows for aimline to be established
• Possible goals
• Level of behavior that is expected several ways
  to establish this
• Norms/percentile cutoffs
• Behavioral expectations
• Realistic/ambitious growth
• Calculated growth rates
• Minimum celeration (Precision Teaching)

31
Goal

• For behavior research indicates that a 75 level
  of performance can be used for non threatening
  behaviors
• For behaviors that are threatening or dangerous a
  100 level of performance should be used

32
Goal

• For example a student that is exhibiting
  difficult maintaining attention to task, the
  intervention plan should be aimed at increasing
  his on task behavior to 75 of the time
• A student that displays dangerous behaviors
  should have an intervention plan that aims at
  increasing appropriate replacement behavior 100
  of the time

33
Goal

• Growth Rate Calculations
• Can be done for any skill, any grade level for
  which norm data is available
• Take a seasonal norm, subtract the earlier
  seasonal norm, and divide by ten (there are ten
  weeks between norming periods)
• For example Winter Norm minus Fall Norm, divided
  by ten weeks
• Gives you a growth expectancy for each week of
  school year
• Allows for obtaining students baseline then
  monitoring progress while comparing to growth
  expectancy

34
Goal

• Growth rate calculation example
• First grade, ID words in sentences
• Winter Norm minus Fall Norm, divided by ten weeks
• 67.9119-35.2694 / 10 3.2643 words per week
• For this reading fluency measure, students are
  acquiring a little over 3 words per week, thus if
  a intervention plan is for six weeks then the
  student should acquire a little over 18 words in
  the six weeks

35
Goal

• Realistic Growth Rates
• Gr 1 2 words/week
• Gr 2 1.5 wrds/week
• Gr 3 1 words/week
• Gr 4 .9 words/week
• Gr 5 .5 words/week

• Ambitious Growth Rates
• Gr 1 3 words/week
• Gr 2 2 words/week
• Gr 3 1.5 wrds/week
• Gr 4 1.1 wrds/week
• Gr 5 .8 words/week

36
Goal

• Celeration rates from the Precision Teaching
  program can be used for goal setting
  consideration
• The lines on the overlay indicate different
  celeration slopes or rates of progress
• Depending on how ambitious the team wants the
  goal to be different rates of progress can be used

37
Minimum Celeration
38
Graph

• Provides a visual representation of a large
  amount of data
• A visual representation of students acquisition
  of skills and allows for easier analysis of
  progress

39
Decision Making Plan

• Facilitates interpretation of data
• Should include
• Rule for raising performance goal 6/4
  consecutive data points above the aimline
• Rule for altering the intervention due to lack of
  progress 3/4 consecutive data points below the
  aimline
• Deno/Allison
• Trend lines

40
Decision Making Plan

• If the decision is to adjust an intervention,
  small changes or refinements are recommended
  before major changes
• However, changes should be substantial enough
  that it has a possibility to result in improved
  student performance
• If making an adjustment, do not make two at the
  same time. It may result in the team being unable
  to determine what caused increased student
  performance

41
Decision Making Plan

• As student performance approaches goal, team must
  decide
• Raise the goal
• Begin work on another target behavior
• Cycle back to previous level/tier

42
Charting Activity

• A means of interpreting large amounts of data
• Allows for relating performance trends to desired
  performance
• Research has shown the charting also facilitates
  positive student outcomes (Fuchs, 1989)

43
Charting Activity

• Definition
• Visual depiction of the students performance
  data, relative to the goal and aimline
• Includes baseline data, goal, aimline, and
  progress monitoring data

44
Charting Activity

• Procedures
• Write the prediction/goal statement
• Depict baseline data collection phase on the
  chart, indicate the median score, identify with a
  heavy dark line
• Depict the goal at the end of the anticipated
  intervention phase
• Depict the aimline by connecting the baseline
  median with the goal, this gives you expected
  rate of progress

45
Charting Activity

• Activity
• Plot the baseline data
• Monday 14
• Tuesday 10
• Wednesday 6
• Thursday 5
• Friday 10
• Plot the median and signify with a heavy dark
  line

46
Charting Activity

• Plot the goal at the end of eight weeks
• 40
• Indicate the aimline

47
Charting Activity

• Performance trends should be analyzed
  periodically
• Trend above aimline raise the goal
• Trend below aimline adjust intervention
• If changes are made to the intervention, indicate
  change on the graph with a squiggle line
• Describe the changes on the back of the chart
• This allows for understanding of specific
  instructional adjustments that were
  successful/unsuccessful

48
Charting Activity

• Plot the first two weeks progress monitoring data
• Week One
• Tuesday 16
• Thursday 14
• Week Two
• Monday 10
• Wednesday 18
• Friday 16
• Make an informed decision regarding the
  effectiveness of the intervention

49
Charting Activity

• Remember
• Rule for raising performance goal 6/4
  consecutive data points above the aimline
• Rule for altering the intervention due to lack of
  progress 3/4 consecutive data points below the
  aimline
• Deno/Allison

50
Charting Activity

• Plot progress monitoring data for next two weeks
• Week Three
• Tuesday 14
• Thursday - 10
• Week Four
• Monday 15
• Wednesday 16
• Friday 16
• Make an informed decision regarding the
  effectiveness of the intervention

51
Charting Activity

• Phase two or three of the intervention?
• Plot the progress monitoring of the next two
  weeks
• Week Five
• Tuesday 24
• Thursday - 26
• Week Six
• Monday 26
• Wednesday 24
• Friday 26
• Make an informed decision regarding the
  effectiveness of the intervention

52
Charting Activity

• Phase two or three of the intervention?
• Plot the progress monitoring of the next two
  weeks
• Week Seven
• Tuesday 28
• Thursday - 30
• Week Eight
• Monday 32
• Wednesday 36
• Friday 38
• Make an informed decision regarding the
  effectiveness of the intervention
• Final decision?

53
(No Transcript)
54
Charting Activity

• Trend lines
• Line that you draw through a series of data
  points that represents the students actual rate
  of progress
• If trend line slope is flatter than aimline
  slope, then adjust intervention
• If trend line slope is steeper than aimline
  slope, then adjust goal
• If slopes are the same, make no change

55
Charting Activity

• Trend line procedures
• Draw vertical line in middle of graph, half data
  points on one side, half on the other if odd
  number of data points, put line through middle
  point
• Draw a vertical line in first half of data to
  separate data half and half
• Draw a vertical line in second half of data to
  separate data half and half

56
Charting Activity

• Trend line procedures
• Draw a horizontal line through median of first
  half of data form an intersection with vertical
  line
• Draw a horizontal line through median of second
  half of data form an intersection with vertical
  line
• Connect the two intersections
• Make your decision, is it the same decision?

57
(No Transcript)
58
Charting Activity

• Technology based charting programs

59
Thoughts and Conclusions

• How can I progress monitor with all the other
  things that I have to do?
• School improvement occurs when focus is placed on
  student improvement outcomes, priorities are
  going to have to be changed so that progress
  monitoring is seen as just as important as
  instruction

60
Thoughts and Conclusions

• We have standards based tests to measure student
  progress, why do more?
• These tests are usually administered once a year
  and show student success in meeting a criterion.
  Progress monitoring enables learning, shows if a
  student is making progress towards success on
  standards based tests, and allows for informed
  instructional decision making that helps the
  student reach the goal of success on standards
  based tests.

61
Formative Evaluation Case Study

• Nicole
• Entered level/tier three in October of her third
  grade year
• Looking at the baseline data, where would you
  intervene?
• What would your goals be?
• Start you charts!

62
Formative Evaluation Case Study
63
Formative Evaluation Case Study
64
Formative Evaluation Case Study

• Nicole
• Progress monitoring data for addition
• Week one 8,3,13
• Week two 13,9,8
• Week three 11,3,11
• Week four 13,12,12

65
Formative Evaluation Case Study

• Nicole
• Progress monitoring data for blends
• Week one 7,8,6
• Week two 6,5,5
• Week three 4,6,4
• Week four 4,4,2
• Week five 6,6,7
• Week six 5,8,10

66
Formative Evaluation Case Study

• Nicole
• Progress monitoring data for sight words
• Week one 61,63
• Week two 64,49
• Week three 51,53
• Week four 50,52
• Week five 56,48
• Week six 44,50

67
Formative Evaluation Case Study

• Nicole
• Based on your progress monitoring data, complete
  the Analysis of Intervention Plan
• Identify median score of last three progress
  monitoring data points for each skill
• Compare current level of performance to
  previously established goals
• Make a decision about the results of the
  intervention plan and the next step
• Is this a student that needs to be considered for
  specially designed instruction?

68
Questions?

• Dr. Tom Jenkins, Director
• Educational Consultation Services, LLC
• Wilmington, NC
• [phone]
• Fanofstel_at_aol.com
• www.educationalconsultationservices.com