The Standards of Learning: An Assessment Approach

1
(No Transcript)
2
" Assessing Assessment  Development,
Implementation, and Effectiveness of the
Standards of Learning "
By John H Spiers
3
Introduction

• A need for a stronger curriculum combined with
  firm accountability
• The basics of the SOL program
• The SOLs and their tests are the future of the
  Virginia educational system

4
Historical Background A Nation At Risk findings

• A cafeteria style curriculum in which the
  appetizers and desserts can easily be mistaken
  for the main courses
• Minimum competencies were maximum expectations

5
A Nation at Risk Recommendations

• Five New Basics
• Curriculum before high school should support the
  Five New Basics
• Improved teacher training and more instructional
  time
• The purpose of testing

6
The Virginia Design for Learning (1980s)

• Why?
• CBE Objectives and SOL Objectives
• SOLs had no assessment tool

7
The Reform of 94

• Raising academic standards
• Developing an assessment tool
• Implementing accountability
• School Performance Report Cards

8
Accountability Factors

• Standard credit
• Verified credit
• School Accreditation
• Full 70 passage on all SOL tests (some
  variance for 3rd grade)
• Accreditation with Warning replaces Provisional
  starting 2003-2004 school year
• Accreditation Denied
• Performance Report Cards

9
General Construction of the SOL Assessment Program

• Each SOL must be rigorous, academic, measurable
  and clear and understandable
• Evaluate minimum knowledge and skills in core
  areas in grades three, five, eight, and EOC
• SOLs are curriculum frameworks

10
Specifics of a Test Blueprint

• Test Development Guidelines
• Blueprint Summary Table
• Expanded Blueprint Summary table

11
Test Development Guidelines

• General Considerations
• Item format
• Supplementary

12
Blueprint Summary Table

• Reporting categories
• Implications of reporting categories
• Omitted SOLs
• Number of field test and operational items

13
Test Construction

• Content Review Committee (CRC) roles
• Students are not notified of which items are
  being field-tested
• Committees reconvene to examine statistical data
  and decide whether or not to use the question

14
Test Bias

• Test bias possibilities
• A separate Bias Committee along with the Content
  Review Committees examine possible test bias

15
SOL Participation of Students with IEP or 504 form

• No accommodations
• Standard accommodations
• Time
• Setting
• Presentation
• Nonstandard accommodations
• Time
• Presentation

16
The Virginia Alternative Assessment Program (VAAP)

• (IDEA) requires assessment in the same core areas
  as the SOLs
• Collection of Evidence - 4 to 6 pieces of
  evidence, at least one must be primary
• Primary source - anecdotal records, work samples,
  performance results surveys, progress and / or
  data charts
• Secondary sources - interviews with the student,
  parent, teacher, and/or employer, pictures, and
  audio-visual communicative devices

17
The VAAP Scoring System

• Determination of student involvement in VAAP
• Rubric for IEP team- scale of one (does not meet
  expectations) to three (fully meets expectations)
  
• Life Skill Strands
• A list of essential SOLs for all students
• The focus of the VAAP

18
Determining Achievement

• Criterion-referenced
• The three achievement levels are
• Does not meet the standards (i.e. fail)
• Proficient in the standards (i.e. pass)
• Advanced achievement in the standards (i.e.
  pass)
• Modified-Angoff Method - multiple choice
• Bookmark Method - English Writing tests

19
Modified-Angoff Method

• For multiple-choice SOL tests
• The method demands diversity
• Three levels of achievement based on percentage
  of 100 barely proficient students or barely
  advanced students that should correctly answer
  the item
• After three ranges of scores have been debated,
  the third range is submitted to the VDOE

20
Bookmark Method

• For SOL English Writing tests
• Diversity in the committee members is crucial
• Committee members place two bookmarks in a
  booklet of multiple-choice items intermingled
  with student writing samples,
• First bookmark- previous material is the
  knowledge and skills of a barely proficient
  student
• Second bookmark- the knowledge and skills of
  a barely advanced student
• The third and final range is sent to the VDOE

21
Other Achievement Level Considerations by the VDOE

• Third, fifth, and eighth grade tests measure
  basics
• Percentage increase in failure rates for every
  one point increase in the passing score
• The VDOE Board selected the highest points of the
  ranges for SOL tests in English, writing, and
  math in elementary and middle school
• Passing scores for high school tests based on the
  complexity and difficulty of content

22
Internal Validity and Reliability (Spring 1998
Administration)

• Spring 1998 Test Administration
• Traditional item statistics
• Rasch item statistics- item difficulty correlates
  to how many people correctly answer it correctly
  and equivalent scores equal equivalent ability
  levels
• Mantel-Haenszel Alpha DIF procedure- probability
  of one group correctly answering a question more
  than another.

23
Internal Validity and Reliability

• Kuder Richardson 20 test of reliability for the
  multiple-choice SOL tests
• Purpose to measure the degree to which test
  questions measure the same body of content and
  skills
• Results
• Person separation reliability for English
  Writing tests.
• Purpose
• Results none included in documentation

24
Validity Testing Through Comparison

• Stanford 9-A broad national test covering
  curriculum non-specific to VA
• Literacy Passport Test in phase-out,
  fundamental skills test
• Results Correlations ranged from 0.54 between
  eight grade SOL Math and sixth grade LPT math to
  .80-.81 between eighth grade SOL English Reading
  / Literature and Stanford 9A Reading

25
The Reality

• SOL tests have high validity to the SOLs and are
  quite reliable

26
Criticisms of the Standards of Learning

• Test results from 2000-2002 present traditional
  academic bias
• Standard complaints for standardized testing (I.
  E. pressure, high-stakes nature)
• Former Principal Corliss Talley- "Any teacher or
  administrator who dared to voice opposition or
  offer criticism of the SOL content, reliability
  or validity was labeled as a troublemaker, a
  renegade or a nonteam player

27
Rebuttals

• Mark Christie notes that one in three Virginian
  students fails the LPT and one in four students
  entering Virginias colleges and universities
  requires remedial work
• SOL test scores from 2000-2002 support the
  contention that student achievement is increasing

28
No Child Left Behind

• Stronger accountability
• Yearly assessment of all students in grades three
  through eight in the areas of reading and math
• NAEP testing
• LEP students to be tested in reading and language
  arts in English after three consecutive years of
  schooling
• School choice
• School choice possible for students in low
  performing schools
• Under-performing schools lose students, part of
  their annual budgets and possibly face
  reconstitution.

29
No Child Left Behind

• Early Reading First
• Freer local control permits states to transfer up
  to fifty percent of its non-Title I state
  activity funds to various ESEA programs without
  advance approval.
• Accountability decisions derived from test scores
  is the future of education

30
Conclusions

• The educational system of Virginia has
  experienced a myriad of reform
• The SOL assessment program is highly valid and
  reliable to the SOLs
• Amidst criticism over SOL testing, standardized
  testing is here to stay
• With No Child Left Behind, many states must
  attempt to catch up to Virginia

31
(No Transcript)