Some Observations on Cognitive Psychology and Educational Assessment

1
Some Observations on Cognitive Psychology and
Educational Assessment
Robert J. Mislevy
University of MarylandNational Center for
Research on Evaluation,Standards, and Student
Testing (CRESST) SMABS Jena, GermanyJuly
18-21, 2004
2
Outline of the talk

• Themes from cog psych
• How cog psych informs what we assess and how we
  might assess it (esp. school work)
• How cog psych helps us understand and organize
  what we do in assessment

3
Themes

• Capabilities limitations
• Reasoning in terms of patterns
• Psychological perspectives
• Acquiring expertise
• Forms of knowledge representation

4
Capabilities limitations

• Ways we are the same / differing / unique
• Experiential reflective cognition
• Optical illusions / cognitive illusions

5
An Optical Illusion (http//www.optillusions.com)
6
Capabilities limitations

• Ways we are the same / differing / unique
• Experiential reflective cognition
• Optical illusions / cognitive illusions
• Limited working memory attention
• Can think about our thinking (metacognition)
• Benefit from procedures, methods, tools

7
Reasoning in terms of patterns

• Perception combines input from environment and
  patterns from experience
• Chi, Feltovich, Glaser example
• Narratives / schemas / scripts / mental models
• This is how we make sense of the world
• Some wired in
• Some learned informally and experientially
• Some through instruction and conscious effort

8
What is this a picture of? (http//www.optillusio
ns.com)
9
Reasoning in terms of patterns

• Simultaneous use of patterns at many levels
• Perception / Meaning /Action
• Key role of interacting with situation
• Inquiry cycle / model-based reasoning
• Interactive tasks (construction, simulation)
• Even in static tasks, focus on perception /
  explanation / action

10
Reasoning in terms of patternsAssessment as
Evidentiary Argument

• What complex of knowledge, skills, or other
  attributes should be assessed ?
• What behaviors or performances should reveal
  those constructs broadly construed?
• What tasks or situations should elicit those
  behaviors?
• (Messick, 1994)

11
Psychological perspectives

• Trait/Differential (Spearman, Carroll)
• Origin of machinery of psychometrics
• Behaviorist (e.g., CRTs of 1970s)
• Developmental (Piaget)
• Information-processing (Newell Simon)
• Sociocultural/situative (Vygotsky, Lave)
• (Greeno, Pearson, and Schoenfeld (1996)

12
Psychological perspectives

• A perspective shapes
• what you pay attention to
• what entities and relationships you use in
  explanations
• what you see as problems and solutions.
• A perspective both enables and constrains
  thinking.

13
Psychological perspectives

• For assessment, perspective shapes
• Inferences you target patterns that shape
  students actions (Meaning Action)
• What you look for in what students say, do, or
  make (Perception)
• What are the features of the situation that evoke
  the evidence you need (Meaning)
• A perspective both enables and constrains what
  you can learn from an assessment.

14
What are the forces at the instant of impact?
20 mph
20 mph

• A. The truck exerts the same amount of force on
  the car as the car exerts on the truck.
• B. The car exerts more force on the truck than
  the truck exerts on the car.
• C. The truck exerts more force on the car than
  the car exerts on the truck.
• D. Theres no force because they both stop.

15
What are the forces at the instant of impact?
10 mph
20 mph

• A. The truck exerts the same amount of force on
  the car as the car exerts on the truck.
• B. The car exerts more force on the truck than
  the truck exerts on the car.
• C. The truck exerts more force on the car than
  the car exerts on the truck.
• D. Theres no force because they both stop.

16
What are the forces at the instant of impact?
10 mph
1 mph

• A. The truck exerts the same amount of force on
  the fly as the fly exerts on the truck.
• B. The fly exerts more force on the truck than
  the truck exerts on the fly .
• C. The truck exerts more force on the fly than
  the fly exerts on the truck.
• D. Theres no force because they both stop.

17
Psychological perspectives

• Hydrive
• Info-processing sociocultural
• AP Studio Art
• Sociocultural interpretational
• Note interpretation of variables in model
• Task-based language assessment
• All perspectives relevant
• Target language use (Bachman Palmer)
• What to stress, how to design situations

18
Acquiring expertise

• Expertise as overcoming human cognitive
  processing limitations
• Patterns for perceiving, understanding, acting
  (incl. sociocultural)
• Use of knowledge representations
• Automating processes to varying degrees
• Metacognitive skills

19
Acquiring expertise

• Examples in assessment
• Katz, re NCARB simulations as example for design
  under constraint (assessment is another such
  domain!)
• Embretson as example for differential perspective
  measurement
• Marshall Derry as example for assessment design
  based on schemata
• Stevens re ordered pairs of actions

20
Forms of knowledge representation

• Symbol sets manipulation
• Forms of knowledge representation (KRs)
• Maps, diagrams, object models, flow charts
• Central to expertise
• Nexus of info-processing sociocultural
• Mediated cognition
• Distributed cognition

21
Forms of knowledge representation

• Some forms of knowledge representation for design
  and using assessments
• Measurement models representations
• Argument structures
• Evidence-centered design structures
• Design patterns, templates, object models
• IMS/QTI standards

22
Three basic models that embody theassessment
argument
Forms of knowledge representation
23
Forms of knowledge representation

• Measurement models
• Multivariate models for different aspects of
  knowledge / skill / propensities (MRCMLM)
• Integration of statistical inference with task
  design (De Boeck, Embretson) Cognitive
  diagnosis, mixed strategies, multilevel models
• Conditional dependence (re interaction)
• Mixed strategies between within persons (Rost
  Huang)

24
Example HYDRIVE

• Student-model variables in HYDRIVE
• Motivated by cognitive task analysis
• Scope shaped by purpose
• Grain-size determined by instructional options
• A Bayes net fragment

25
HYDRIVE, continued

• A Bayes Net Measurement Model, docked with
  Student Model

Library of Measurement Model fragments
26
Conclusion

• Assessment is a particular kind of narrative
• An evidentiary argument about aspects of what
  students know and can do, based on a handful of
  particular things that have said, done, or made.
• Assessment integrates perceiving, understanding,
  and acting.
• Assessment forms both enable and constrain
  thinking about students.

27
Conclusion

• Cognitive psychology helps us understand what to
  make inferences about, what we need to see, what
  situations can provide us with clues.
• Conceiving targets of assessment
• Explicating and improving the design and use of
  assessments