How do we know what they know

1
How do we know what they know?

• FASS Meeting
• Orlando, FL
• April 5, 2004
• Arthur Eisenkraft (eisenkraft_at_att.net)

2
Are we listening?

• The optometrist
• The Duracell competition
• Two objects falling in a vacuum
• The cord of wood

3
No Child Left Behind

• Enormous concern about NCLB and other high stakes
  assessments.
• NCLB a potential nightmare
• AYP - need of improvement
• public embarrassment
• best students leaving and the scores dropping
  more, closing of schools.
• What can we do?
• This year alone 26,100 of the nations 91,400
  have been labeled schools that need
  improvement. (Sam Dillon, 1 in 4 Schools Fall
  Short Under Bush Law, N.Y. Times, January 27,
  2004 at A21)

4
NCLB - Advice

• Assessment in Support of Instruction and
  Learning Bridging the Gap Between Large-Scale
  and Classroom Assessment - Workshop Report
  (2003)Board on Testing and Assessment,
  Mathematical Sciences Education Board, Center for
  Education
• www.nap.edu

5
The Deborah Meier Amendment

• A basic test should first be taken by the folks
  we honor by electing to office.
• The people who legislate or mandate a test should
  be required first to take it themselves to ensure
  that it's measuring what they think it is. It's a
  form of validity checking.
• They might even have their scores posted!
• Seeking Alternatives to Standardized Testing
• By Jay Mathews, Washington Post Staff
  WriterTuesday, February 17, 2004 1046 AM

6
Todays Discussion

• Formative classroom assessment can positively
  impact instruction and therefore is our best
  approach to students performing better on all
  tests.
• What can teachers do?
• What can we do to support teachers?

7
Brief History of Assessment

• When did it all begin?
• No Child Left Behind
• FCAS
• New York State Regents

8
(No Transcript)
9
(No Transcript)
10
How do we assess?

• Please discuss
• Observation
• Portfolio
• Projects
• Questioning
• Paper and pencil
• Interview
• Presentation
• Checklist
• Skills
• Self assessment
• Quizzes
• conferences

11
Classroom Assessment

• The Grade Book
• Tests
• Quizzes
• Homework
• Class participation (?)
• Lab reports
• Attendance (X)
• Projects
• The Final Exam
• Local
• State High stakes
• These are often treated as summative though they
  do inform as formative.
• Other formative assessments include
• Questions in class
• Practice tests

12
Get tests back immediately

• They can then be used for formative assessment.
• How can anyone continue instruction when you have
  a tool that informs you of student understanding?
• Easily measured by supervisors and students alike

13
Formative Assessment

• The value of formative assessment (Paul Black)
  students often have limited opportunities to
  understand or make sense of topics because many
  curricula have emphasized memory rather than
  understanding. Textbooks are filled with facts
  that students are expected to memorize, and most
  tests assess students abilities to remember the
  facts.

14

• What Goes Wrong?
• Tests that do not correlate with understanding
• Force Concept Inventory (FCI)
• Regents exam question on moving galaxies
• Private Universe videotapes

15

• What Goes Wrong?
• Tests that do not correlate with understanding
• Force Concept Inventory (FCI)
• Regents exam question on moving galaxies
• Private Universe videotapes
• Were not testing what we teach
• Harris cartoon of mouse and maze

16
(No Transcript)
17

• What Goes Wrong?
• Tests that do not correlate with understanding
• Force Concept Inventory (FCI)
• Regents exam question on moving galaxies
• Were not testing what we teach
• Harris cartoon of mouse and maze
• Were not teaching what we test
• Waldo phenomenon

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23

• Improvements
• Rubrics
• Clearly defined grading schema matrix
• B and A-----
• Have students help create rubric
• Ownership
• Motivation
• Have students self evaluate with rubric

24
The grading rubric /-

• Student Grade Teacher Grade
• A
  A
• A C
• C
  C

25

• Improvements
• Rubrics
• Clearly defined grading schema matrix
• B and A-----
• student and teacher comparisons A,A or C,C or
  A,C or C,A
• all require very different discussions
• Saphier effective instruction
• testing for understanding
• how do you know what the students know?

26
Cognitive Empathy

• With references from
• The Skillful Teacher
• Jon Saphier

27
Checking for Understanding

• Knowing when students dont understand suggests
  that teachers have means for checking for
  understanding.
• What means do we have for checking for
  understanding?

28
Checking for understanding

• Presses on
• No test return for 3 weeks
• After math lesson, heres your 25 problems
• Take a clean sheet, were going on
• No clue there are kids in the room
• Never asks students to explain
• Incorrect response, - can anyone else answer

29
Checking for understanding

• Presses on
• Reads cues
• Their looks
• Eye contact
• Nodding heads
• Asleep or awake
• Misbehavior
• I can see it in their eyes

30
Checking for understanding

• Presses on
• Reads cues
• dipsticks
• White-boarding
• Short quiz
• Raise the hand
• Choral answers
• Cards with A or B
• Raise fingers with 1(index) or 2
• List answers on board which answer is the best

31
Checking for understanding

• Presses on
• Reads cues
• dipsticks
• Uses recall questions
• What do we already know
• List examples
• Who invented
• Wheres waldo
• Definitions
• Name the parts of the microscope
• The scientific method
• Restating what is already known

32
Checking for understanding

• Presses on
• Reads cues
• dipsticks
• Uses recall questions
• Uses comprehension questions
• Explain
• Justify
• Compare
• Apply
• Calculate
• Why
• Summarize

33
Checking for understanding

• Presses on
• Reads cues
• dipsticks
• Uses recall questions
• Uses comprehension questions
• Anticipates confusion
• Photosynthesis, Krebs cycle
• Understanding that some kids are literal
• Underground railroad
• Misconceptions research
• Look at prior knowledge
• Teacher examining their own assumptions

34
A TEST for Checking for Understanding

• How do you know that a student understands?
• What evidence do you have?
• How often should you be able to answer this
  question?

35
The National Science Education Standards
(NSES) Less Emphasis On Assessing what is
easily measured More Emphasis On Assessing
what is most highly valued Less Emphasis On
Assessing to learn what students do not
know More Emphasis On Assessing to learn what
students understand
36
Instructional Models

• Karplus
• three-phrase learning cycle
• exploration, invention and discovery
• Lawson
• exploration, term introduction, and concept
  application
• Bybee ? 5E
• Engage, explore, explain, elaborate, evaluate
• 7E clarification of 5E

37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
4 Q Assessment Model

• What does it mean?

43
4 Q Assessment Model

• What does it mean?
• How do we know?

44
4 Q Assessment Model

• What does it mean?
• How do we know?
• Why should I believe?

45
4 Q Assessment Model

• What does it mean?
• How do we know?
• Why should I believe?
• Why should I care?

46
The other four questions

• What did you say?
• Should we take notes?
• When is class over?
• Will this be on the test?

47

• Challenges
• Identify who are we testing
• Students
• Teachers
• Schools and districts

48
Challenges

• Identify for what purpose
• (from Classroom Assessment and the NSES, NRC)
• Help students learn
• To illustrate and articulate the standards for
  quality work
• To inform teaching
• To guide curriculum selection
• To monitor programs
• To provide a basis for reporting concrete
  accomplishments to interested parties
• For accountability
• Certification
• Reporting individual achievement
• Grading
• Placement
• Promotion
• Accountability
• Parents to taxpayers
• (from High Stakes Assessments, NRC)

49
Challenges

• Are we trying to use ONE instrument
• for all (students, teachers, schools)?
• for all purposes?
• Understanding vs. belief
• Mazur student taking FCI

50
www.nap.edu
51
The Assessment Triangle

• cognition, observation, and interpretationmust
  be explicitly connected and designed as a
  coordinated whole. If not, the meaningfulness of
  inferences drawn from the assessment will be
  compromised.

C O
I
52
Assessment 1

• Question What was the date of the battle of the
  Spanish Armada?
• Answer 1588 correct.
• Question What can you tell me about what this
  meant?
• Answer Not much. It was one of the dates I
  memorized for the exam. Want to hear the others?

53
Assessment 2

• Question What was the date of the battle of the
  Spanish Armada?
• Answer It must have been around 1590.
• Question Why do you say that?
• Answer I know the English began to settle in
  Virginia just after 1600, not sure of the exact
  date. They wouldnt have dared start overseas
  explorations if Spain still had control of the
  seas. It would take a little while to get
  expeditions organized, so England must have
  gained naval supremacy somewhere in the late
  1500s.

54
Comparison

• Most people would agree that the second student
  showed a better understanding of the Age of
  Colonization than the first, but too many
  examinations would assign the first student a
  better score.
• When assessing knowledge, one needs to understand
  how the student connects pieces of knowledge to
  one another. Once this is known, the teacher may
  want to improve the connections, showing the
  student how to expand his or her knowledge.

55
Formative Assessment Research

• Black and Wiliam (1998) provide an extensive
  review of more than 250 books and articles
  presenting research evidence on the effects of
  classroom assessment.
• They conclude that ongoing assessment by
  teachers, combined with appropriate feedback to
  students, can have powerful and positive effects
  on achievement.
• They also report, however, that the
  characteristics of high-quality formative
  assessment are not well understood by teachers
  and that formative assessment is weak in
  practice. High-quality classroom assessment is a
  complex process

56
Return to bad practice

• We revert back to old, failed strategies when
  under pressure
• Comfort food
• In dealing with our children
• Youll poke your eye out
• What if everybody jumped off the Empire State
  Building?
• Prison, drug addiction

57
Responding to the pressure (NCLB, FCAS)

• If you go back to the worksheets and it doesnt
  work, you wont be blamed in the same way because
  this is what everybody does.
• If you continue with something that is closer to
  the edge, you are more open to criticism
• This could be the reason for the blame
• Fail forward as Thomas Edison did

58
Support our teachers

• Help them to not succumb to the pressures.
• Encourage them to improve their instruction, not
  to revert to poor pedagogy.
• Give them permission to be better teachers.

59
Questions that foster deep understanding rather
than questions that ask for repetition of
memorized information and conclusions. (Jim
Minstrell and Emily van Zee)

• Ask or promote questions to open an inquiry and
  elicit students initial understanding and
  reasoning.
• Ask or promote questions to interpret and make
  sense of data in order to generate new knowledge
  and understanding.
• Ask or promote questions to clarify or elaborate
  on observations and inferences.
• Ask or promote questions to encourage learners to
  justify their answers and conclusions or to
  explain their reasoning to go beyond a mere
  stating of an answer.
• Ask or promote questions to extend or apply
  learned ides.
• Ask or promote questions that help learners
  monitor their own learning.

60
Understanding by Design

• When do we generate test questions
• UbD (Grant Wiggins, Jay McTighe)
• Enduring understandings
• Evidence
• Instruction

61
Not in vain

• When studying about veins and arteries, for
  example, students may be expected to remember
  that
• arteries are
• thicker than veins,
• more elastic, and
• carry blood from the heart
• veins carry blood back to the heart.

62
Sample test item

• Arteries
• a. Are more elastic than veins
• b. Carry blood that is pumped from the heart
• c. Are less elastic than veins
• d. Both a and b
• e. Both b and c

63
The new science of learning

• does not deny that facts are important for
  thinking and problem solving.
• Research on expertise in areas such as chess,
  history, science,and mathematics demonstrate that
  experts abilities to think and solve problems
  depend strongly on a rich body of knowledge about
  subject matter (e.g.,Chase and Simon,1973Chi et
  al.,1981deGroot,1965).

64
Facts are not enough

• However,the research also shows clearly that
  usable knowledge is not the same as a mere
  list of disconnected facts.
• Experts knowledge is
• connected and organized around important concepts
  (e.g., Newton s second law of motion)
• conditionalized to specify the contexts in
  which it is applicable
• supports understanding and transfer (to other
  contexts) rather than only the ability to
  remember.

65
Vein and artery experts

• Know the facts in the mc question
• also understand why veins and arteries have
  particular properties.
• They know that blood pumped from the heart exits
  in spurts
• That the elasticity of the arteries helps
  accommodate pressure changes.
• They know that blood from the heart needs to move
  upward (to the brain) as well as downward and
  that the elasticity of an artery permits it to
  function as a one-way valve that closes at the
  end of each spurt and prevents the blood from
  flowing backward.
• They are better able to transfer
• Design an artificial artery strong enough to
  handle pressure with or without elasticity
  (Bransford and Stein,1993).

66
Classroom environments

• Formative assessmentsongoing assessments
  designed to make students thinking visible to
  both teachers and students are essential.
• They permit the teacher to
• grasp the students preconceptions,
• understand where the students are in the
  developmental cor-ridor from informal to formal
  thinking, and
• design instruction accordingly.
• In the assessment-centered classroom environment,
  formative assessments help both teachers and
  students monitor progress.

67
Learner friendly assessments

• Provide students with an opportunity to revise
  and improve their thinking (Vye et al.,1998b),
• help students see their own progress over the
  course of weeks or months
• help teachers identify problems that need to be
  remedied (problems that may not be visible
  without the assessments).
• Problem based learning model
• Sport for the moon

68
Defining Science Content

• Facts, process, knowledge
• Order of the planets
• I went to a two day history seminar and there was
  no content I didnt learn any new dates.
• Where is the knowledge we have lost in
  information. (T.S. Eliot)

69
Wait time study at Kings College (Black
Wiliam, 2000)

• Questions opened up discussion that helped expose
  and explore students assumptions and reasoning.
• At the same time, wrong answers became useful
  input, and the students realized that the teacher
  was interested in knowing what they thought, not
  in evaluating whether they were right or wrong.
• As a consequence, teachers asked fewer questions,
  spending more time on each.

70
Wait time study at Kings College (Black
Wiliam, 2000)

• In addition, teachers realized that their lesson
  
  
  
  
  
  
  
  
  
  
  planning had to include careful thought
  about the selection of informative questions.
• They discovered that they had to consider very
  carefully the aspects of student thinking that
  any given question might serve to explore.
• This discovery led them to work further on
  developing criteria for the quality of their
  questions.

71
Grades vs. comments

• Simply giving grades on written work can be
  counterproductive for learning (Butler, 1988)
• In response, teachers began instead to
  concentrate on providing comments without
  gradesfeedback designed to guide students
  further learning.
• Students also took part in self-assessment and
  peer-assessment activities, which required that
  they understand the goals for learning and the
  criteria for quality that applied to their work.
• In these ways, assessment situations became
  opportunities for learning, rather than
  activities divorced from learning.

72
Sliding toward Socrates

• Follow-up research
• Grades vs. comments vs. grades comments
• No distinction between instruction and assessment

73
Beliefs and Practice



Your text and program 1 2
3 4
1 2 3 4
What you value (e.g. inquiry, content)
74
Beliefs and Practice



Your text and program 1 2
3 4
1 2 3 4
What you value (e.g. telling and memorizing)
75
Beliefs and Practice



Your text and program 1 2
3 4
1 2 3 4
What you value
76
Beliefs and Practice


o
o
o o
Your text and program 1 2
3 4
1 2 3 4
What you value (e.g. inquiry, content)
77
Beliefs and Practice
o o
o

Your text and program 1 2
3 4
1 2 3 4
What we value (e.g. telling and memorizing)
78
Beliefs and Practice



Our text and program 1 2
3 4
1 2 3 4
Content, pedagogy, assessment factors
79
Summary

• Return tests promptly
• Rubrics
• Understanding by Design
• Enduring undestandings, evidence, instruction
• Testing for undestanding
• Know what each student knows at the end of every
  class
• 7E learning model
• Assessment triangle Cognition, Observation,
  Interpretation
• Dont revert to non-productive behavior
• Define science content to include inquiry and
  process
• Facts in meaningful context

80
Assessment in the movies

• Do we recognize this quality of assessment?

81
In Conclusion

• The best way in which to deal with NCLB is to
  improve instruction in the classroom.
• This can be done with improvements in formative
  assessment
• Formative assessment has been shown to be
  effective at improving student achievement.
• Help teachers learn Testing for understanding
  techniques.
• Provide teachers with the confidence and support
  to continue to improve their teaching and resist
  the temptation to revert to poor pedagogy and
  worksheets in attempts to deal with NCLB.
• Give them permission to be good teachers.

82
Common Advice

• .the wisdom to know what I can change
• My husband handles the important issues

83

• We must be the change we want to see in the
  world.
• - Gandhi

84
(No Transcript)
85
(No Transcript)
86
Get inside a students head.

• Cognitive empathy for the workings of the
  learners minds -- an ability to put themselves
  in the learners shoes

87
Unscrambling Confusion

• Determining what students dont understand
  implies that teachers have ways of unscrambling
  confusions that identify the specific point(s) of
  misunderstanding and deal with them.
• What means do we have for unscrambling
  confusions?

88
Unscrambling Confusions

• None

89
Unscrambling Confusions

• None
• Re-explains

90
Unscrambling Confusions

• None
• Re-explains
• Isolates point of confusion with pinpoint
  questions

91
Unscrambling Confusions

• None
• Re-explains
• Isolates point of confusion with pinpoint
  questions
• Perseveres and returns

92
Unscrambling Confusions

• None
• Re-explains
• Isolates point of confusion with pinpoint
  questions
• Perseveres and returns
• Has student explain own current thinking

93
Three pillars of assessment

• a model of how students represent knowledge and
  develop competence in the subject domain
• tasks or situations that allow one to observe
  students performance,
• an interpretation method for drawing inferences
  from the performance evidence thus obtained.

94

• Improvements
• Rubrics
• Clearly defined grading schema matrix
• B and A-----
• student and teacher comparisons A,A or C,C or
  A,C or C,A
• all require very different discussions
• Eliciting prior understandings
• Fish is fish

95
(No Transcript)
96
(No Transcript)
97
Standards
State exams (good instruction)
(good preparation)