Expertise Studies and Assessment

1
Expertise Studies and Assessment

• Robert J. Mislevy
• University of Maryland
• February 19, 2001
• draws on Ericsson Smith, 1991

2
Studies of Expertise

• Basic ideas
• Cognitive Task Analysis (CTA)
• CTA for assessment
• NCARB example

3
Basic Ideas (1)

• The study of expertise seeks to understand and
  account for what distinguishes outstanding
  individuals in a domain from less outstanding
  individuals in that domain.
• One would expect social evaluations to be greatly
  influenced by observations of previous
  performances (not all by the same individual)
  occurring under unique circumstances.
• Differential approach proved unsatisfactory.

4
Basic Ideas (2)

• Focus on producing and observing outstanding
  performance under reproducible and relatively
  standardized conditions.
• Aim to analyze and describe the cognitive
  processes critical to producing outstanding
  performance on such tasks.
• Examine critical cognitive processes, and account
  for explicit learning mechanisms to account for
  their acquisition.

5
Cognitive Task Analysis (1)

• A traditional job analysis focuses on valued
  tasks in a domain, in terms of how often people
  must perform them and how important they are.
• A cognitive task analysis (CTA) focuses on the
  knowledge people use to carry out those tasks.

6
Cognitive Task Analysis (2)

• A CTA in a domain seeks to shed light on
• essential features of the situations
• internal representations of situations
• the relationship between problem-solving behavior
  and internal representation
• how the problems are solved and
• what makes problems hard.
• (Newell Simon, 1972)

7
Potential Observations
t0 Start of info presentation for specific task
t1 Generation of answer
Post-process observations Memory for task
info Retrospective report Post-experiment
interview
8
Some Pervasive Results

• Knowledge structures are key, for
• Perceiving situations
• Solving problems in terms of generative
  principles of the domain
• Generating plans monitoring progress
• Automaticity
• Frees resources for novel aspects of problems
• Extensive reflective practice, with feedback.

9
CTA for Assessment

• CTA can help ground assessment, re validity
  argument and principled/explicated task
  construction.
• Must aim to specify elements of Student-,
  Evidence-, and Task-models,
• (vs. cognitive scientists aims!)
• in light of the intended assessments purpose,
• and constraints of the assessment-building and
  assessment-delivery contexts.

10
NCARB Example (Irv Katz, ETS)

• Design under constraints
• Planning the bank site
• Differences between novices and experts
• Success rate 98 vs. 88
• Planning time execution time
• Patterns of revision
• Implications for assessment
• Nature of constraints
• Rationale