Establishing Credibility Evidence for Teacher Work Samples

1
Establishing Credibility Evidence for Teacher
Work Samples

• Training for Scoring
• Benchmarking
• Reliability
• Validity
• Impacts on PK-12 Student Learning

2
Training for Scoring

• Review Teaching Processes and Standards
• Study TWS Prompt and Scoring Rubrics
• Agree on Road Map of Evidence
• Develop Indicator Definitions
• Conduct Anti-Bias Training
• Practice Scoring

3
Benchmarked Performances

• Proto-typical performances at each level of the
  developmental scoring rubric.
• 1 Beginning
• 2 Developing
• 3 Proficient
• 4 Expert
• Benchmarks are used for training of scorers and
  mentoring of candidates.

4
TWS Holistic Score Distribution January 2002

• Beginning 18
• Developing 48
• Proficient 29
• Expert 5
• n 135

5
Inter-Rater Reliability Total Analytic
Scores

• Extent to which scoring decisions are consistent
• across expert judges using the analytic rubric.
• Dependability Coefficients
• (Shavelson Webb, 1991)
• 5 raters .88
• 3 raters .82
• 1 rater .60

6
Inter-Rater Reliability Analytic
Sub-Scales
Sub-Scale
5 Raters
3 Raters
1 Rater
Contextual Factors
.86
.79
.55
Learning Goals
.85
.77
.53
Assessment Plan
.86
.78
.54
Design for Instruction
.86
.79
.55
Instructional Decision-Making
.78
.78
.42
Analysis of Student Learning
.82
.74
.48
Reflection and Self-Evaluation
.86
.79
.56
7
Inter-Rater Reliability
Trained raters are able to score with sufficient
inter- rater agreement to make high-stakes
decisions about performance levels. Moreover,
scores are sufficiently dependable that
reliable decisions can be made for each
teaching process separately.
8
Validity
Expert judgments of . . .

• Content Representativeness (Crocker,
  1997)
• -- Realism
  -- Criticality
  -- Necessity
  -- Frequency
• Alignment with INTASC Principles
  

9
Validity Realism
98.1 of the raters strongly agreed with the
statement The performance tasks required by
the TWS represent actual lessons delivered to
students in PK-12 school classrooms.
10
Validity Criticality
97.5 of the expert judges rated the tasks
required by the TWS as critical or important
to teaching.
11
Validity Frequency
All of the tasks required by the TWS were rated
by the expert judges as high frequency activities
for teachers 90 of more of the raters said
weekly or daily for all of the tasks.
12
Validity Necessity
Overall, does the TWS assess knowledge and skills
that are necessary for beginning
teachers? Absolutely 70 Yes 30 No 0
13
Alignment with INTASC Principles
INTASC Principle
Directly
Implicitly
Not at All
5
45
50
Subject matter
50
5
45
Human development and learning
2
8
95
Adapting instruction
20
2
78
Instructional strategies
7
50
43
Classroom motivation and management
30
58
12
Communication skills
2
3
95
Instructional planning skills
0
7
93
Assessment of student learning
7
40
53
Professional commitment/responsibility
Partnerships
55
38
7
14
Alignment with INTASC Principles
INTASC Principle
Directly
Implicitly
Not at All
5
45
50
Subject matter
50
5
45
Human development and learning
2
8
95
Adapting instruction
20
2
78
Instructional strategies
7
50
43
Classroom motivation and management
30
58
12
Communication skills
2
3
95
Instructional planning skills
0
7
93
Assessment of student learning
7
40
53
Professional commitment/responsibility
Partnerships
55
38
7
15
Impacts on Student Learning

• Quality of the sources of evidence (learning
  goals and assessments).
• Percent of students who met the learning goals.
• Percent of students who showed increased learning
  (improvement).

Evidence of student learning is situated within
the context of the quality of the learning goals
and assessments.
16
Student Learning Index

• Dependability coefficients for 3 raters .71.
• Highest ratings were for the quality of the
  learning goals (e.g., challenging, significant,
  and appropriate).
• Lowest ratings were for assessment criteria are
  clearly explained and reliable and assessments
  directly measure the learning goals.
• Correlations between ratings on Student Learning
  Index and analytic scores on TWS .89.

These data have important implications for
teacher education program curriculum.