The Role of Comprehension in HumanComputer Skill Acquisition

1
The Role of Comprehension in Human-Computer Skill
Acquisition

• Stephanie Doane
• Mississippi State University
• Department of Psychology

2
Comprehension A Paradigm For Cognition

• Kintsch (19881998) Construction-Integration
  Model
• Psychological process theory of mental processes
  involved in comprehension
• Characterizes comprehension process as constraint
  satisfaction.
• Comprehension occurs when and if the elements
  that enter into the process achieve a stable
  state in which the majority of the elements are
  meaningfully related to each other and other
  elements that do not fit the pattern of the
  majority are suppressed.

3
Construction
Hippos have legs
Hippos color gray
Nick exists
Hippos live on land in water
Nick color gray
Manatees color gray
Manatees live in water
Nick lives in water
Parrots color gray
Parrots fly
Nick is gray in color and lives in water.
4
Integration (1)
Hippos have legs
Hippos color gray
Nick exists
Hippos live on land in water
Nick color gray
Manatees color gray
Manatees live in water
Nick lives in water
Parrots color gray
Parrots fly
Nick is gray in color and lives in water.
5
Integration (2)
Hippos have legs
Hippos color gray
Nick exists
Hippos live on land in water
Nick color gray
Manatees color gray
Manatees live in water
Nick lives in water
Parrots color gray
Parrots fly
Nick is gray in color and lives in water.
6
Comprehension A Paradigm For Cognition

• Kintsch (19881998) Construction-Integration
  Model
• Psychological process theory of mental processes
  involved in comprehension
• Characterizes comprehension process as constraint
  satisfaction.
• Comprehension occurs when and if the elements
  that enter into the process achieve a stable
  state in which the majority of the elements are
  meaningfully related to each other and other
  elements that do not fit the pattern of the
  majority are suppressed.

7
Cognitive Processes as Constraint Satisfaction

• Decision making
• Explanatory coherence (Thagard 1998-present)
• Analogy
• Analogical mapping (Holyoak Thagard
  1989-present)
• Comprehension
• Understanding text (Kintsch 1988-present)

8
Comprehension Its not just for reading anymore!

• Planning
• Routine tasks (Mannes, Kitijama Polson 1991
  1995)
• Novel tasks (Doane, Sohn, Kintsch, Clawson
  McNamara Polson 1989 1992 1997)
• Battlefield Planning (Barnes, Sohn, Doane to
  appear
• Skill Acquisition
• Learning from a tutor (Doane, Sohn, McNamara in
  press)
• Visual Attention
• Pilot visual attention (Doane Sohn in press)

9
How Do Users Acquire Interactive Skills?
Role of User Comprehension in Interactive Task
Performance
Methods
Tasks
Empirical Studies
Cognitive Models
10
Methodology
Develop User Model
Validate User Model
Design Recommendations
Inform the Theory
11
Methodology
Develop User Model
then
Knowledge Processing Assessment
Model Construction
Review Performance Data
Build Idealized Knowledge Base
Cognitive Task Analysis
Build Individual Knowledge Bases
12
Building a Theoretical Model of UNIX Expertise

• Review UNIX user performance data
• Cognitive task analysis to determine knowledge
  required to produce commands
• Construct a knowledge base for modeling
  prototypical "expert" using Kintsch's theory of
  comprehension

13
UNIX Command Production Skill

• Understanding Performance Deficits
• Types of Commands
• Singles
• Do X on A
• Example SORT A
• Multiples
• Do X on A, Do Y on B, Do Z on C
• Example SORT A, TAIL B, LPR C
• Composites
• Do X on A, then Do Y on A', then Do Z on A''
• Example SORT FILE TAIL LPR
• (Note pipe redirecting input and output)

14
UNIX Performance Data

• Subjects computer science majors
• Group UNIX Experience
• Novices (10) 6 months to 1 year
• Intermediates (10) 1.5 years to 3 years
• Experts(10) 3 years plus.
• Results Percent correct productions
• NoviceltIntermediateltExpert
• Composites a problem for each group!
• Conclusion Knowledge of single elements in a
  command language are not sufficient to combine
  the elements together in a novel fashion

15
(No Transcript)
16
Cognitive Task Analysis

• Single Multiple Commands
• Knowledge
• Command Syntax
• Memory
• Command Executed

• Composite Commands
• Knowledge
• Command Syntax
• Input and Output Concepts
• Input and Output Syntax
• Command Input and Output
• Memory
• Command Executed
• Order of Commands
• Intermediate Results

Knowledge and memory prerequisites greater for
composites
17
Validating Cognitive Task Analysis

• Empirical Prompting Study -- where prompts aid
  with the knowledge/memory prerequisites
  hypothesized
• Modeling individuals in the prompting study --
  predicting actions performed in response to
  prompts
• Comparing modeled and actual individual
  performance as a function of prompt

18
EMPIRICAL PROMPTING STUDY

• Goal
• Descriptive validation of cognitive analysis and
  user model
• Approach
• Give UNIX users composite tasks, and provide them
  with assistance if they fail to produce the
  correct command
• If knowledge such as command syntax is lacking, a
  prompt providing command syntax knowledge should
  assist performance
• If the locus of the performance deficit is
  related to memory load, then a prompt intended to
  reduce working memory load should assist
  performance
• 22 UNIX users (N, I, E), 21 composite tasks tasks

19
(No Transcript)
20
Greatest change for novices following ordering
prompt
21
Knowledge Scores by Prompt
Effect of I/O Conceptual
Effect of Command Syntax
Effect of Command Redirection Conceptual
Effect of I/O Syntax
22
EMPIRICAL PROMPTING STUDY

• Results
• Novices and Intermediates -
• Problems ordering the elements in a composite
  command
• Need ordering prompts to obtain perfect
  performance.
• Experts
• Attain almost perfect performance prior to any
  ordering prompts.
• Conclusions
• Supports analysis -
• chaining elements together requires relating the
  pieces of knowledge in an ordered fashion and
  tracking the intermediate results in working
  memory (Sohn Doane 1997)

23
Model Validation Predicting individual user
learning

• Model how individuals "learn" from prompts
• Background
• Case-Based Learning Storage and retrieval of
  examples (Hammond,1989 Mannes, 1991)
• Search-Based Learning Chunking results of a
  search process used to solve a problem (e.g.,
  SOAR Rosenbloom, et al., 1991)
• Analogy-Based Learning Analogical interpretive
  processes (ACT-R, Anderson, 1998 Holyoak
  Thagard, 1996)
• Comprehension-Based Learning Governed by the
  context-sensitive and dynamically changing
  activation of interassociated knowledge (Doane et
  al.,in press, Cognitive Science Schmalhofer,
  1993)

24
Comprehension-Based Learning

• Mechanism
• Activation of prompted knowledge
• Activation constrained by associative
  relationships between world and preexisting
  knowledge
• Transfer of prompt propositions from temporary
  world knowledge to permanent knowledge
• Prompt must be retained in world knowledge and
  satisfy a precondition of plan element considered
  for firing in order to transfer
• Consequence is use in subsequent interactions
• (use following learning 100 in model, 96 for
  users)

25
Comprehension-Based Learning

• Modeling memory constraints
• Prompts retention based on context-sensitive
  working memory limitations (four for all)
• If a prompt is dropped from working memory before
  it can satisfy a precondition, it is not
  transferred to permanent knowledge - it isnt
  learned
• Adding correct knowledge
• If prompt satisfies fired plan element
  precondition it is learned
• Deleting incorrect knowledge
• Contradictions of learned prompts deleted

26
Validate User Model
Descriptive Validation
Predictive Validation
then
Examine group data Perform CTA
Examine subset of individual performance data
Build train group prototype models
Build and test individual models
Compare individual modeled performance
Compare group modeled performance
27
Validation Method

• For each subject modeled (22), develop initial
  knowledge base by scoring knowledge displayed
  without prompting
• Give model the same "problem statements given to
  subjects
• If the model fails to produce the correct
  command, then give it prompt. (Prompts provide
  preconditions for plans, and if plan activated,
  will now fire.)
• Examine match between subject and model data
  using group and individual results to obtain
  goodness of fit measure

28
Example Knowledge Representations

• Type of Knowledge Propositional Representation
• World Knowledge
• File exists in directory (EXIST INDIRECTORY
  FILE)
• General Knowledge
• Command Syntax
• nroff formats file (KNOW NROFF FORMAT FILE)
• Plan Knowledge
• Name
• format contents of a file (DO FORMAT FILE)
• Preconditions
• know nroff formats file (KNOW NROFF FORMAT
  FILE)
• know nroff -ms flag (KNOW NROFF TAKESFLAG
  -MS)
• know file exists in directory (KNOW INDIRECTORY
  FILE)
• Outcome(s)
• formatted file exists (KNOW FORMATTEDFILE
  ONSCREEN)

29
Construction Relationships

• Within declarative propositions
• argument overlap
• embedding
• (exists A B) (exists B C) 1 argument overlap
• (exists A B C) (exists AB) 1 embedded
  proposition
• Causal chaining between plan elements

30
Constructed Connectivity Matrix
31
Procedure Modeling Individual User Performance
32
Procedure Data Reduction and Analysis

• Same scoring rules for model same as for users
• Root-mean-squared deviations (RMSDs) based on
• Percent correct fit Number of prompts required
  by actual and corresponding modeled user to
  produce correct composite
• Knowledge score fit Amount of knowledge
  displayed in each production attempt by model and
  user as a function of prompt
• Error fit errors displayed by modeled and actual
  users as a function of prompt
• How similar
• is accuracy as a function of prompt?
• are learning patterns as a function of prompt?
• are errors as a function of prompt?

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Summary

• Comprehension-based theory accounts for a
  significant amount of user performance when
  learning from technical instructions

37
Theoretical Contributions

• Extend comprehension-based theory to explain and
  predict what users will attend to, understand,
  use and then learn to improve performance in a
  training environment.
• Suggests the centrality of comprehension-based
  processes in planning and learning.

38
Design Contributions

• Model can be used to focus designer's attention
  on user knowledge and memory prerequisites
• Results can be used to modify existing system
  design.
• Sohn Doane (1997) Tested design recommendations
  for graphical interfaces for UNIX. For novices,
  graphical aids must reduce working memory demands
• Implications for computer-aided tutoring
• Student model component

39
Ongoing Research Visual Attention

• Predict
• What airplane pilots will attend to and what they
  will find difficult about accomplishing complex
  sequence-dependent flight maneuvers, such as
  turning and changing speed
• Performing Maneuvers
• Requires the comprehension of task instructions
  and information displayed in the cockpit and
  association of this information with piloting
  knowledge in order to develop action plan.
• Focus
• is on the way cockpit information activates
  piloting knowledge, how this dictates what
  displays and controls pilots attend to, and how
  it influences action plans .

40
Questions?