SAS?EVAAS?

1
SAS?EVAAS?

• A Way of Measuring Schooling Influence
• Marty Ward and Dana Wrights
• Winston-Salem/Forsyth County Schools

2
SAS?EVAAS?

• Unites two respected entities
• 1. SAS a major software company that produces
  powerful data analysis capabilities
• SAS is the worlds largest privately held
  software company.

3
SAS?EVAAS?

• 2. EVAAS is Educational Value-Added Assessment
  System.
• EVAAS is based on more than ten years of research
  by Dr. William Sanders and his colleagues on
  value-added assessment.

4
SAS?EVAAS?

• EVAAS was provided to all North Carolina LEAs by
  the Department of Public Instruction in 2007-08
• Current tests included in EVAAS are End-of-Grade
  Reading and Math, all End-of-Course subjects,
  Writing 4, 7, and 10, and SAT (school level only)

5
What is value-added assessment?

• It is a different approach to analyzing student
  achievement data.
• It is based on the simple notion of following a
  students academic progress over time.
• By linking each students test records from grade
  to grade over subjects, the influence of the
  district, school and teacher on the rate of
  academic progress can be extracted via complex
  data analysis.

6
What is value-added assessment?

• Much of the value-added research looks at the
  impact of the teacher.
• Although reports can be made based on teacher,
  the North Carolina package does not include
  teacher-level value-added reports.
• However, the State Board of Education funded the
  addition of teacher-level reporting.

7
Value-added Assessment

• Historically, accountability reporting is based
  on a snapshot of student achievementproficiency
  or performance composite.
• SAS EVAAS methodology extracts information from
  longitudinally linked student test scores.

8
Value-added Assessment

• ABCs growth is the closest available measure to
  EVAAS
• ABC growth looks at a students past performance
  a maximum of two tests and expects relative
  performance to be at least as good as in the past
• EVAAS uses up to 5 years of test scores in every
  available subject (at least 3 scores) to predict
  performance

9
Limitations in Using EVAAS

• Students school is where they were last
  testednot re-rostered (at the current time)
• Another data tool, not perfect
• Like any good data tool, it is more likely to
  raise questions rather than provide answers

10
Limitations in Using EVAAS

• Less useful for grade 3 (limited data)
• Less useful when there are small numbers of
  students
• Less reliable estimates
• Cannot reasonably disaggregate
• Scores from alternate assessments not robust
  enough for use (not included in EVAAS)

11
SAS?EVAAS? Advantages

• The EVAAS methodology minimizes the influence of
  measurement error by using up to five years of
  data for an individual student. Analyzing all
  subjects simultaneously increases the precision
  of the estimates.

12
SAS?EVAAS? Advantages (cont.)

• By including all students in the analyses, even
  those with a sporadic testing history, SAS EVAAS
  provides the most realistic estimate of
  achievement available for a district or school.

13
SAS?EVAAS? Advantages (cont.)

• SAS EVAAS allows educators to benefit from all
  tests, even though their scales are different.
  The EVAAS methodology accommodates all tests
  that
• are reliable
• are highly correlated with curricular standards
• have sufficient stretch in the reporting scale to
  measure the achievement of both very low- and
  very high-achieving students in a grade or
  subject.

14
SAS?EVAAS? Advantages (cont.)

• With SAS EVAAS methodology, each student serves
  as his or her own control, creating a level
  playing field and eliminating the need to adjust
  for race, poverty, or other socioeconomic
  factors. This innovative approach ensures that
  the results are fair to both student and
  educators.

15
EVAAS Reflecting Back and Looking Forward

1. Students score histories are used to provide
   valuable diagnostic information about past
   practices.
2. Each students previous scores are used to
   predict success probabilities of numerous
   academic milestones (future EOG and EOC scores).

16
Value-Added Report

• How effective was the school/district?
• How much progress did students make compared to
  the average effectiveness of North Carolina
  schools?

17
(No Transcript)
18
(No Transcript)
19
Value-Added Report

• This is the main type of EVAAS report. It has the
  most rigorous statistical standards.
• The value-added reports are conservativefor a
  school or district to be labeled as having
  progress significantly below or significantly
  above average progress in the state, it must be
  2 standard errors below or above.

20
Value-Added Report Utility

• Given how students were predicted to perform if
  they received the average North Carolina
  schooling experience, how effective were we?
• Pinpoints overall strengths and weaknesses by
  subject and grade - provides FOCUS

21
Diagnostic Report

• Divides students into 5 groups based on how they
  were predicted to perform.
• (Quintiles the bottom 20 in NC, the next
  lowest 20, the middle group, and so on).
• Graphic shows how effective the school/district
  was with each of the groups.
• Can also look at subgroups, e.g. ethnic or gender
  groups.

22
(No Transcript)
23
(No Transcript)
24
Diagnostic Report

• Green line is reference line the amount of
  progress students were predicted to make
• Bar above the line students progressed better
  than average
• Bar below the line students made less than
  average progress
• Red vertical line 1 standard error above below

25
Diagnostic Report Utility

• Given the rest of the state as a comparison, how
  were our students predicted to perform
  (quintiles)?
• How did our students actually perform compared to
  students with similar performance predictions?
• Were there differences for various subgroups?

26
Diagnostic Performance Reports

• Very similar to Diagnostic Reports, but divides
  students into predicted Achievement Levels (Level
  I, II, III, or IV)
• Graphic shows how effective the school was with
  each of the levels.
• Can also look at subgroups, e.g. ethnic or gender
  groups.

27
(No Transcript)
28
(No Transcript)
29
Diagnostic Performance Report Utility

• Given the rest of the state as a comparison, how
  were our students predicted to perform (via
  achievement levels)?
• How did our students actually perform compared to
  students predicted to perform at the same
  achievement level?
• Were there differences for various subgroups?

30
Diagnostic Reports Utility

• Drill-down from Value-added
• Targets which subgroups and students need the
  most intervention
• Finds success stories
• SHOULD NEVER BE USED FOR ACCOUNTABILITY ONLY FOR
  DIAGNOSTIC PURPOSES

31
At Risk Reports

• Students at risk to not be at grade level (AYP At
  Risk), probability between 0 and 70 for math,
  0 to 80 for Reading (because of new standards).
• Graduation At Risk (0 to 70 for courses
  required for high school graduation)
• Other at risk (other EOC tests)

32
(No Transcript)
33
Student Pattern List

• Provides a list of students based on selected
  subject, grade, course
• Allows user to further select a subgroup, if
  desired
• Divides the group into thirds (lowest, middle and
  highest) and estimates effectiveness

34
(No Transcript)
35
(No Transcript)
36
Custom Student Reports

• Provides ability for user to create and save
  rosters of user-defined subgroups of students.
• For example, you can define a group likely to be
  successful in the next grade or course for
  planning purposes

37
Custom Student Reports

• Allows a quick search for
• Individual students
• Groups by selected characteristics (e.g. gender,
  ethnic group, LEP, predicted scores)
• Allows drill down to show 5 years of student
  percentile scores (compares to school and
  district)

38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Custom Reports Utility

• Flexibility to build school or teacher-level
  reports (last year or possibly current)
• Sort capabilities can provide
• Prioritized list of need
• Quick grouping of AYP groups

42
Custom Reports Caution

• Probabilities are calculated based on the premise
  that NO INTERVENTION OCCURS beyond the average NC
  schooling experience
• Should NOT be treated as a score simple
  adjustments to a students education plan can
  make tremendous improvements to probability

43
Student Profile Reports

• Click on a student name
• Provides line graph of students previous
  performance in selected subject in terms of
  percentiles
• Includes district and school averages for
  comparison purposes

44
(No Transcript)
45
Student Profile Reports Utility

• Quick graphic view of trends
• Parent conferences
• Comparison to school and district

46
Student Projection Reports

• Click on a students probability
• Provides line graph of students previous
  performance in selected subject in terms of
  percentiles, PLUS a yellow projection line of
  next test
• Includes projected proficiency level average
  percentile for comparison

47
(No Transcript)
48
Student Projection Reports Utility and Caution

• Provides student probability of meeting various
  academic milestones
• Based on previous performance AND IF NO
  INTERVENTION OCCURS beyond the average NC
  schooling experience
• Caution should be exercised when sharing with
  student and/or parent must carefully explain
  meaning of probability

49
Things to Remember

• Use the Help Link
• Dont let it overwhelm you
• Consider this tool an investigative project you
  will learn more each time you enter the site
• Improvements are coming
• Some things are difficult to manipulate right now
  (e.g., rerostering), but SAS EVAAS is working to
  improve the site

50
Things to Remember

• Best use is a diagnostic tool
• This is ONE of MANY tools available for
  evaluating data to improve instruction
• Use with other tools to get a clear picture
• Goal Summaries
• District Assessments
• Formative Assessments

51
WEBSITEncdpi.sas.com/evaas/login.jsp