Prepared for the

1
Moving Students Beyond the Curve Assessment
that Works!

• Prepared for the
• Select School-Based Assessment Team of
• Williamsburg-James City County Public Schools
• by Dan Mulligan, Ed. D.
• December 2007

2
Terminology

• Educational Test
• is a formal attempt to determine a students
  status with respect to specific variables, such
  as the students knowledge, skills, vocabulary,
  and attitude.
• Note 1 test and assessment are
  interchangeable, as long as test is not
  restricted to traditional paper-and-pencil
  experiences.
• Note 2 measurement and examination are also
  in the same synonym pool.

3
Pre-assessment

• Each team member should be supplied with an index
  card.
• On the index card, write the grade level or
  course that you currently are teaching.
• Think about the process (suggestion write a
  bulleted draft) you engage in to create an item
  that appears on your assessment.
• On the index card, write a synthesis of that
  process
• Relaxyou have 5 minutes.

4
Key Questions for Authoring a Meaningful
Assessment
What is it we want all students to learn? How
will we know if they have learned it? How will
we respond when they dont?

• Without continuous growth and progress, such
  words as improvement, achievement, and success
  have no meaning.
• Benjamin Franklin

5
Evaluating Suffolks BEST!!!
CAUTION Are there any peanut allergies in the
house?
6
(No Transcript)
7
WJCC (05-06) Graduation Rate All 80 Black
58 Hispanic 75 White 86 Disability
42 Poverty 57 LEP ---
Required pass rates for each subgroup 2005
tests reading (65)
math (63) 2006 tests reading (69) math
(67) 2007 tests reading (73) math
(71) 2008 tests reading (77)
math (75)
8
Self Reliance
There are three types of baseball players--those
who make it happen, those who watch it happen,
and those who wonder what
happened. Tommy Lasorda
teachers administrators
9
Opportunity to Learn
Has the strongest relationship with student
achievement of all school-level factors. What
Works in Schools, ASCD

• Three types of curricula were identified by SIMS
  TIMS
• The Intended Curriculum content/skill specified
  by the state, division, or school at a particular
  grade level.
• The Implemented Curriculum content/skill
  actually delivered by the teacher.
• The Attained Curriculum content/skill actually
  learned by the students.

Implemented Curriculum
Attained Curriculum
Intended Curriculum
10
Content-Related Evidence of Validity(Intended
Curriculum)
Essential Skills
Essential Knowledge
Learning TARGET (content validity)
Essential Vocabulary
11
Essential Knowledge
Essential Vocabulary
Essential Skills
What is the result of pushing an object?
What does CURVED mean?
How far will the block travel? Why?
12
What is pi?
What is the formula for circumference of a circle?
Create a situation involving area of a circle.
Essential Vocabulary
Essential Knowledge
Essential Skills
13
The BIG Picture
What does biennial mean?
What are the responsibilities of the legislative
branch?
Use the given table to explain the similarities
and differences in the responsibilities of the
branches of government
Essential Vocabulary
Essential Knowledge
Essential Skills
14
Your Turn
Essential Knowledge Essential Skills Essential
Vocabulary
15
(No Transcript)
16
The value of teacher teams analyzing student
achievement data is dependent on the validity and
reliability of the instrument used to generate
the data.
17
The Validity of a Test(Validity of Score-Based
Inferences)

• Tests, themselves, do not possess validity.
• Validity refers to the accuracy of the inferences
  about students status with respect to the
  assessment domain.
• There are three types of validity evidence to
  consider for any assessment
• Content Related The extent to which an
  assessment procedure adequately represents the
  content of the assessment domain being sampled.
• Criterion Related The degree to which
  performance on an assessment procedure accurately
  predicts the examinees performance on an
  external criterion.
• Construct Related The extent to which empirical
  evidence confirms that an inferred construct
  exists and that a given assessment procedure is
  measuring the inferred construct accurately.

18
Content Related Validity
What message does each item communicate about
the focus of instruction for this particular SOL?
In other words, what must each student understand
about this SOL based on the item.

• Assessment begins the instruction/learning
  process
• SOL Students will comprehend the formal and
  informal nature of the interrelationships among
  the executive, legislative, and judicial branches
  of the federal government.

• Sample Item 2
• Which, if any, of the following statements about
  governmental stability is true? (mark each
  statement as True or False.)
• The enactment of term-limiting legislation at
  the local level has made the U.S. federal
  legislative branch of government more stable.
• The availability of the impeachment process tends
  to decrease the stability of the executive branch
  of U.S. government.

Sample Item 3 Our founding fathers charted a
meaningful series of governmental checks and
balances. Focus on the area of taxation, then
select two of the three branches and briefly
describe the formal way(s) in which one branch
can check the other. _______________________ ____
___________________ _______________________ ______
_________________

• Sample Item 1
• Which of the following three branches of the
  U.S. government, if any, is primarily responsible
  for the final enactment of treaties with foreign
  nations?
• Legislative
• Executive
• Judicial
• No single branch is responsible

FOCUS Students will need to learn why important
factors such as governmental stability are
present for each branch.
FOCUS Students will need to learn the formal
and informal nature of the relationships among
the governmental branches.
FOCUS Students will need to learn the primary
responsibilities of each branch.
19
Grade 3 Science
What message does each item communicate about the
focus of instruction for this particular SOL? In
other words, what must each student understand
about this SOL based on the item.
20
Content-Related Evidence of ValidityYour Turn

• Select a standard
• Create 2 items such that each item addresses a
  unique instructional focus area.
• Write the focus area for each item on the back of
  sheet.
• You have 10 minutesenjoy!!!!

21
Limitations of Paper-and-Pencil Tests

• Inefficiently measures creative types of learning
  targets
• Requires a long amount of time to develop
  especially if designing high-quality items from
  scratch
• Susceptible to bias in test items , making the
  test invalid for some or all students

22
Designing High-Quality Paper-and-Pencil Tests
Handout pages 5 - 12

• Parts of a Multiple-Choice Item
• Directions
• Visual
• Item Stem
• Response
• Options
• Stimulus
• (Not pictured. This can be
  a phrase, quote or a short narrative that
  the student has to read.

• Use the picture below to answer the following
  question.
• About how many paper clips long is this pencil.
• A 2 paper clips
• B 3 paper clips
• C 4 paper clips
• D 5 paper clips

23
Principles for Writing Quality Multiple Choice
Items

• Align and organize your assessment items with the
  state standards by -
• Identifying both the CONTENT and the LEVEL OF
  THINKING that your item will address.
• Applying the Principles for Developing Quality
• Items Stems
• Response Options
• Stem Starters
• Visuals in Test Items
• Item Format to Align with State Assessments

24
Principles for Writing Quality Item Stems

• Write the item stem in one of two ways.
• As a good, solid question
• As a statement, that when matched with the
  correct answer, is a complete thought
• Overall, state item stems positively.
• However, students need practice with items that
  use negative words (e.g. NOT, LEAST, IS NOT,
  EXCEPT, NOT TRUE). These words should be
  emphasized using bold or italics.
• Stuff the stem.
• Include most of the information for your question
  in the stem and not in the responses.

25
Principles for Writing Quality Response Options

• Provide brief responses.
• Have one correct answer.
• Write plausible distracters.
• Arrange numbers in ascending or descending order.
• Write the answer choices so they are
  grammatically consistent with the stem.
• Make sure the right answer is always there.
• Watch trick key words. Examples include ALL,
  ALWAYS, ENTIRE, EVERY, NEVER, NONE, ONLY,
  COMPLETELY.
• Arrange answer choices vertically.

26
Stem Starters

• Stem Starters are content neutral item stems that
  contain blanks for you to insert appropriate
  content.
• Stem Starter Which word has the same vowel sound
  as ____?
• 1. Which word has the same 2. Which word does
  NOT have the vowel sound as cat? same vowel
  sound as cat?
• A box A bat
• B bag B bag
• C pot C hat
• D pin D pin

27
Principles for Using Quality Visuals

• Include visuals on some of your test items.
• Use a variety of visual representations
• Make sure each visual representation has a
  purpose.
• Place visuals in different parts of the item.
• Make sure visuals are clearly reproduced and easy
  to read.
• Use more than one question for complex graphics.
• Place questions related to the visual on the same
  page or facing pages.

28
Refine and Apply Understanding of FORMAT Issues

• Revisit the items developed earlier.
• Check and revise items to target
• Stem construct
• Plausibility of distracters
• You have 10 minutesenjoy!!!
• Pairs Share

29
Criterion-Related Validity

• The degree to which performance on an assessment
  procedure accurately predicts the examinees
  performance on an external criterion.

30
Sample Organizing Topic from the Virginia Science
Enhanced scope Sequence Guide
31
e
d
b
32
(No Transcript)
33
Types of Assessment Items and Formats Related to
Different Aspects of Learning
Assessments
Transforming Classroom Grading, ASCD, 2002
Note L Low, M Medium, H - High
34
e
d
b
35
Writing and Thinking

• If students are to make knowledge their own,
  they must struggle with the details, wrestle with
  the facts, and rework raw information and dimly
  understood concepts into language they can
  communicate to someone else. In short, if they
  are to learn, they must write.
• National Commission on Writing,
  2003

36
(No Transcript)
37
p
YOUR TURN! Work with a partner to align each item
to a target objective. Hint Look for knowledge,
skills, and vocabulary.
n
L
g
38
(No Transcript)
39
a
YOUR TURN! Work with a partner to align each item
to a target objective. Hint Look for knowledge,
skills, and vocabulary.
c
f
b
40
(No Transcript)
41
b,e
b,f
YOUR TURN! Work with a partner to align each item
to a target objective. Hint Look for knowledge,
skills, and vocabulary.
b,g
42
YOUR TURN

• Begin with the END in MIND
• Select an assessment target from a content sheet.
• Create an assessment to validly measure each
  students understanding of the specified skill,
  knowledge and vocabulary.
• Complete step 2 for at least 3 learning
  objectives.
• Please work in teams of no more than 2 or 3
  people.
• Please be prepared to justify explain your
  work with another team.
• Enjoy

43
Collecting QUALITATIVE Data
44
Thinking Inside The Box Recording Sheet
45
Mix it Up in the Box

• Listen for the topic and the amount of time
• Silently mix around the room
• When when directed pair up with person closest to
  you
• In pairs, Partner A shares and Partner B listens
• Partner B responds to what he/she heard by
  paraphrasing LET ME TELL YOU WHAT I UNDERSTOOD
  YOU TO SAY
• Switch Roles

46
Thinking inside the Box
What is an action taken by you, your department
or your school that you attribute to resulting in
increased achievement for each student impacted
by the action this year?
What use of common assessments currently exists
in your grade-level. How often? Do
teachers/students analyze results, then
reteach/relearn reassess when necessary?
Look at the picture above. How does this picture
relate to assessment? Complete this sentence The
image is like assessment in that_______
During the assessment instruction process do
teachers focus on essential knowledge, essential
vocabulary and essential skills (or some but not
all)?
47
Create teams of not more than 4 people. For each
question One person in the team take the lead by
sharing with your team either your response or
the response obtained from your travels. Other
team members can then share their findings.
What is an action taken by you, your department
or your school that you attribute to resulting in
increased achievement for each student impacted
by the action this year?
What use of common assessments currently exists
in your grade-level. How often? Do
teachers/students analyze results, then
reteach/relearn reassess when necessary?
Look at the picture above. How does this picture
relate to teaching? Complete this sentence The
image is like assessment in that_______
During the assessment instruction process do
teachers focus on essential knowledge, essential
vocabulary and essential skills (or some but not
all)?
48
A pupil from whom nothing isever demanded
which hecannot do, never does all he can.
John Stuart Mill
No one Rises to Low Expectations. Carl Boyd
49
WE ASSESS TO

• INFORM INSTRUCTIONAL DECISIONS
• 2. ENCOURAGE STUDENTS TO TRY

50
CLASSROOM ASSESSMENT

• DECISION?
• WHO?
• HELPFUL INFO?

• What comes next in the learning?
• Students, Teachers Parents
• CONTINUOUS info on how each INDIVIDUAL student is
  doing on her/his JOURNEY TO each standard

51
Formative Assessment

• Formative assessment is the process used by
  teachers and students during instruction that
  provides feedback to adjust teaching and learning
  for the purpose of improving student learning.

Council of Chief State School Officers, October
2006
Notes Process rather than a particular
test. It is not the nature of the test itself
that makes it formative or summativeit is the
use to which those results will be put.
52
Talk to Me

• Directions
• Form a team of EIGHT (8) people
• Determine the person with the most sisters and
  then send them to pick-up your team ziplock bag
  PLEASE DO NOT OPEN!!!
• Determine the person with the least sisters and
  send them to pick-up a grid sheet for each
  person.
• Distribute a grid sheet to each team member.

53
(No Transcript)
54
Follow-up Debriefing

• Each pair should share with your other team
  members the method you used to graph the figure.
• Discuss with your team
• Which method appeals to you?
• Is there another method that you would prefer?
• Prepare for a pairs choice of method with a new
  graph.

55
(No Transcript)
56
Guiding Self-Evaluation Questions

• Self evaluation
• thinking about what happens when we are
  learning
• (Choose one and add the words of the learning
  intention)
• What really made you think while you were
  learning to ?
• What helped you (e.g. friend, the teacher, new
  equipment, a book, your own thinking) when
  something got tricky about learning to
• What do you need more help with about learning
  to ?
• What are you most pleased with about learning to
  ?
• What have you learned that is new about (quote
  learning intention)?
• How would you change this activity for another
  group/class who were learning to ?

57
Formative assessment Often referred to as
assessment for learning, formative
assessment ...refers to all those activities
undertaken by teachers, and by the students in
assessing themselves, which provide information
to be used as feedback to modify the teaching and
learning activities in which they are engaged.
Such assessments become formative when the
evidence is actually used to adapt the teaching
to meet the needs. Black and Wiliam (1998)
58
Defined

• Formative assessment
• Often referred to as assessment for learning,
  formative assessment
• ...refers to all those activities undertaken by
  teachers, and by the students in assessing
  themselves, which provide information to be used
  as feedback to modify the teaching and learning
  activities in which they are engaged. Such
  assessments become formative when the evidence is
  actually used to adapt the teaching to meet the
  needs.
• Black and Wiliam (1998)

59
(No Transcript)
60
Student opportunity to reflect on what was clear
and what was confusing Plan of Action Clear
message to the student that this is a
beginningnot an end Discussion question Are
assessments considered a pit stop of learning
or an end of learning in your school/class?
61
The Formative Assessment Process
62
Learning Goals

As a result of what we do today I will create
a story question involving fractions in the
solution. solve a story question from one of my
team members.
Student Understand the target. (Focus on
learning goals.) Teacher Select and clearly
communicate the learning target (understand
typical learning progressions in the area.)
63
Research-Based Elements of Formative Assessment

• Prior Knowledge Assessment
• Feedback
• Teaching and Assessing for Transfer
• Explicit Criteria
• Self Assessment

64
Four Levels of Formative Assessment

• Teachers Instructional Adjustments
• Students Learning-Tactic Adjustments
• Classroom-Climate Shift (transforms a
  traditional, comparison-dominated classroom into
  an atypical, learning-dominated classroom)
• School-wide Implementation (teacher learning
  community)

65
Teacher/Student Conference to Agree on
Improvement Strategies
66
Teacher/Principal or Teacher/PLC Conference to
Agree on Improvement Strategies
67
Department or Grade Chair/Principal or PLC
Leader/PLC Group Conference to Agree on
Improvement Strategies
68
Key elements of Formative Assessment include

• The identification by teachers and learners of
  learning goals, intentions or outcomes and
  criteria for achieving these.
• Rich conversations between teachers and students
  that continually build and go deeper.
• The provision of effective, timely feedback to
  enable students to advance their learning.
• The active involvement of students in their own
  learning.
• Teachers responding to identified learning needs
  and strengths by modifying their teaching
  approach(es).

69
Essential Attributes of Formative Assessment

• Sufficient Time The information from formative
  assessments must be available in sufficient time
  for (1) teachers to adjust how theyre teaching
  what was assessed or (2) students to adjust how
  theyre trying to learn what was assessed.
• Actual Adjustments Assessment-based adjustments
  must actually take place in a teachers
  instructional activities or in students learning
  activities during the instructional segment in
  which the assessment took place.
• Targeted Students Whatever adjustments are made,
  by teachers and/or by students, must be intended
  to enhance the learning of the students who were
  assessed.

70
Organizing Theme Things someone would say
The Holiday Edition
Santa Claus
200 POINTS
Dr. Mathews
a principal
100 POINTS 100 POINTS
Hillary Clinton
a student
a teacher
50 POINTS 50 POINTS 50 POINTS
71
(No Transcript)
72
Spotlight Moment
Instructional Strategy Regular checking and
evaluation of student understanding of the
learning goal throughout the lesson.
Please number off at your table.
Pair off using the rule odd number next even
number (e.g., 12, 34, 56, )

• Discuss your observation of
• teacher understanding of the effect on learning
  and

73
What is higher order thinking?

• "Every day thinking, like ordinary walking, is a
  natural performance we all pick up. But good
  thinking, like running the l00-yard dash, is a
  technical performance... Sprinters have to be
  taught how to run the 100-yard dash good
  thinking is the result of good teaching, which
  includes much practice."
• David Perkins, Howard University

74

• Thinking Skills Test
• What kind of grollers were they?
• 2. What did the grollers do?
• 3. Where did they do it?
• 4. In what kind of gak did they grangle?
• 5. Place one line under the subject and two lines
  under the verb.

MoralStudents can answer low-levelquestions
without thinking. Students enter/exit
classroomswith no more understanding ofwhat
they've learned than"The Griney Groller"taught
you!
75
How do YOU know?
76
(No Transcript)
77
Commitment
If you dont make a total commitment to whatever
youre doing, then you start looking to bail out
the first time the boat starts leaking. Its
tough enough getting that boat to shore with
everybody rowing, let alone when a guy stands up
and starts putting his life jacket on. Lou
Holtz
78
Human nature, if it is healthy, demands
excitement and if it does not obtain its
thrilling excitement in the right way, it will
seek it in the wrong. God never makes bloodless
stoics He makes no passionless
saints. Oswald Chambers
Wishing you new passion in your schools
instructional program. Dan Mulligan, Simply
Achieve, Inc.
79
It's a WRAP!!!
80
People without information can not act. People
with information can not help but act.
81
Teamwork??
People without information cannot act. People
with information cannot help but act. Ken
Blanchard
82
(No Transcript)
83
(No Transcript)
84
(No Transcript)
85
(No Transcript)
86
(No Transcript)
87
(No Transcript)
88
(No Transcript)
89
(No Transcript)
90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
(No Transcript)
101
(No Transcript)
102
(No Transcript)
103
(No Transcript)
104
(No Transcript)
105
(No Transcript)
106
(No Transcript)
107
(No Transcript)
108
(No Transcript)
109
(No Transcript)
110
(No Transcript)
111
(No Transcript)
112
(No Transcript)
113
(No Transcript)
114
(No Transcript)
115
(No Transcript)
116
(No Transcript)