Classroombased Assessment and Technical Quality

1
Classroom-based AssessmentandTechnical Quality

• Can they exist and work together?

2

• Panel Doug Christensen
• Commissioner of Education Nebraska
• Barbara Plake
• Former Director of Buros Center for Testing
• Margaret McInteer
• Director of Professional Development-ESU 4
  Nebr.
• Moderator
• Pat Roschewski
• Director of Statewide Assessment - Nebraska

3
Why would a state policy support
classroom-based assessment? What does it take
to assure the technical quality of
classroom- based assessment? How is this
implemented?
4
Why ? Decisions about whether or not students
are learning should be made where the learning is
taking place the classroom. 2000
5
President Congress U.S. Dept of Educ.
An External Assessment System
Governor State Board of Educ. Commissioner Nebr.
Dept of Educ.
Local Board Superintendent Teachers Students
Does not engage the most important stakeholders.
6
Places the most important stakeholders in the
center.
7
Important Consideration The technical quality
should be focused on the use of the data
obtained, not just the instrumentation itself.
8
Important Consideration The validity of the
inference not just the test.
9
What ? What does it take to assure the technical
quality of classroom-based assessment?
10
Mixing Technical Quality with Classroom
Assessments Oil and Water?

• Barbara S. Plake

11
Use of Classroom Assessment to Enhance Learning

• Substantial learning gains by strengthening use
  of classroom assessments (Black Wiliam)
• Three key features
• Providing accurate information
• Providing high quality feedback
• Involving students in the assessment process

12
Need for accurate information

• Clearly linked to learning targets
• Clear and appropriate purpose
• Match between assessment target and assessment
  method
• Clearly written
• Minimize bias
• Scoring is appropriate and accurate

13
Technical Quality for Assessments

• Principles of Technical Quality
• Validity can the results of the assessment be
  used for the intended purposes?
• Reliability can the results of the assessment
  be considered a reasonable representation of the
  students knowledge and skill?

14
Linking Technical Quality to Classroom Assessments

• Essential for the positive gains in student
  achievement to be realized
• Not just applicable to large-scale assessments
• Technical principles apply to ALL assessment
  types and uses

15
Applying Validity to Classroom Assessments

• Validity can the results of the assessment be
  used for the intended purposes?
• Ensuring that the test questions are aligned with
  the learning targets (content standards)
• Ensuring that the test questions are congruent
  with instruction (opportunity to learn)
• Ensuring that student performance on the test is
  due to their achievement and not affected by
  irrelevant factors (minimize bias)

16
Applying Validity to Classroom Assessments

• Validity can the results of the assessment be
  used for the intended purposes?
• Ensuring that the results can be interpreted as
  indicating what students know and are able to do
• Ensuring that the decisions made from the test
  results are appropriate (score decisions mastery
  level cutscores)

17
Applying Reliability to Classroom Assessments

• Reliability can the results of the assessment
  be considered a reasonable representation of the
  students knowledge and skill?
• Are the students results trustworthy?
• Can the results be generalized to other similar
  tests on the same content?

18
Applying Reliability to Classroom Assessments

• Reliability can the results of the assessment
  be considered a reasonable representation of the
  students knowledge and skill?
• Are the results consistent with other indicators
  of the students achievement?
• Are the results repeatable?

19
Technical Quality Indicators for Classroom
Assessment

• Validity linking the preparation, development,
  administration, scoring, and interpretations to
  the purposes of the assessment.
• Do the test scores represent what was intended?
• Do the test results provide accurate and useful
  information?

20
Threats to Validity in Classroom Assessments

• Does the test appropriately represent the content
  and instructional emphasis?
• Test blueprint should cover the content
  appropriately
• Testing method should allow for meaningful
  interpretations
• Confounding factors should be eliminated (reading
  level, background knowledge, etc.)

21
Threats to Validity in Classroom Assessments

• Are the test questions clear and well written?
• Do the students understand what they are being
  asked to do?
• Do the students know what will be scored?
• Are there hints in the test questions that could
  allow for correct response without reaching the
  learning target?

22
Threats to Validity in Classroom Assessments

• Is the administration appropriate?
• Do the students have adequate time to complete
  the test questions?
• Can the results be clearly interpreted as
  representing the students achievement?
• Copying, cheating
• Group work

23
Threats to Validity in Classroom Assessments

• Scoring
• Do the number of score points support the
  intended interpretations?
• Is there a clear correct or best answer to the
  question?
• Is the scoring defensible and not capricious or
  arbitrary?
• Is the scoring fair?

24
Threats to Validity in Classroom Assessments

• Interpretations of test results
• Are grading decisions appropriate?
• Are mastery levels set using appropriate
  procedures?
• Can the intended interpretations be supported?

25
Threats to Reliability for Classroom Assessments

• Has the student shown the ability to do the test
  tasks in other settings?
• Will the format of the test affect the students
  ability to demonstrate achievement?
• Is there something about this assessment that
  helps or hinders the students ability to do well
  on the test?
• Is there sufficient breadth of difficulty
  represented on the test so weaker and stronger
  students can demonstrate their achievement levels?

26
Threats to Reliability in Classroom Assessments

• Is the scoring dependable and trustworthy?
• Can the results be generalized beyond these
  specific tasks and questions?

27
Mixing Classroom Assessments and Technical Quality

• Most of the work that teachers do in developing
  their tests is consistent with good testing
  quality
• Many teachers dont realize that there are
  building tests with good indicators of validity
  and reliability
• Technical measurement jargon interferes

28
Need for technical quality in classroom
assessments

• Test results are used in classroom settings
• Communicate student achievement
• Make instructional decisions for students
• Evaluate instructional effectiveness
• Make diagnostic decisions for students
• Reporting achievement results for accountability
  purposes

29
Need for Technical Quality

• If the classroom assessments do not have the
  requisite level of technical quality, none of the
  identified uses are justifiable or appropriate
• Teachers are the kingpin in this setting
• Know the students
• Make the decisions
• Deliver the instruction

30
Teachers Role in Classroom Assessment

• Teachers should demand access to high quality
  information for their teaching
• It isnt possible to deliver high quality
  instruction without high quality information
• High quality assessments empower teachers to do
  their job!

31
Oil and Water?

• Good teaching and good testing go hand-in-hand
• More like mixing fine wine with pure Rocky
  Mountain spring water!

32
How ? How is this implemented?
33
Top Ten Reasons The Nebraska Model WorksHow We
Do It
Nebraska

• Margaret McInteer
• ESU4

34
Reason Number Ten

• We believe in our process
• ( high quality, locally developed assessments in
  combination with national tests and a state
  writing exam)

35
Reason Number Nine

• Our Assessments Match Our Standards and Theyre
  Free from Bias
• Each district must determine that the assessment
  measures the standards and that the students have
  sufficient opportunity to demonstrate their
  ability to meet the standards. They also must be
  free from bias

36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Reason Number Eight

• Our Model is upside down

40
The classroom is at the top of our model

• We believe that impacting student learning begins
  and ends in the classroom, so we put it at the
  top of our model

41
Reason Number Seven

• Our students have the opportunity to learn and
  the level is appropriate

42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
Reason Number Six

• Examining Data Has Changed Our Way of Doing
  Business

46
(No Transcript)
47
Reason Number Five

• Our teachers are our secret weapon
• (With limited funds, we chose to put our money
  into our teachers)

48
(No Transcript)
49
Reason Number Four

• Why go out for hamburger when youve got steak at
  home?
• The Buros Institute is in our backyard

50
From the beginning

• Weve had a partnership with the Buros Institute
  to ensure quality assessments

51
Reason Number Three

• Our assessments must meet reliability
  requirements
• We use appropriate processes for setting mastery
  levels

52
(No Transcript)
53
(No Transcript)
54
Reason Number Two

• Whod A Thunk?
• Our teachers really do understand the technical
  aspects of assessment

55
Reliability

• We use Decision Consistency to determine
  reliability for objectively scored assessments.
• And inter-rater reliability for subjectively
  scored assessments

56
We use modified Angoff to determine mastery

• Teachers create Proficiency Level Descriptions
  for each standard. Focusing on the barely
  progressing, barely proficient, and barely
  advanced, teachers make judgments on how target
  students will perform on each item.

57
(No Transcript)
58
Reason Number One

• It really is all about our kids

59
(No Transcript)
60
Questions?