The Growth of Cognitive Modeling in HCI Since GOMS

1
The Growth of Cognitive Modeling in HCI Since GOMS
Anna Kolesnichenko Songmei Han
2
Overview

• GOMS as cognitive modeling
• Advances in modeling specific serial components
• Extensions of the basic framework
• What Cognitive Modeling in HCI can and cannot do

3
Cognitive modeling

• the progress in modeling the kind of cognition
  involved in HCI
• basic and advanced sets of parameters
• account for the time of given activities
• formal modeling in grammars and production
  systems
• Error production
• Time to learn
• Savings from previous learning
• critical path analysis
• Specification of interacting processes and their
  durations

4
Cognitive Models

• Predict how users will interact with proposed
  designs
• Constrain the design space
• Answer specific design decisions
• Estimate total time for task performance
• Provide base for calculating training time and
  designing training documentation
• Determine stages of activity that take the
  longest time or produce the most errors

5
Cognitive modeling (cont.)

• Method used for design, evaluation and training
• Gaps in understanding the process of interacting
  with computers
• Human learning
• Design of consistent user interfaces
• Error production and management
• Interpretation of visual displays for meaning
• Concurrent vs. sequential processes

6
Gaps in cognitive theory

• Fails to capture
• Users fatigue
• Individual differences
• Mental workload
• Change expected in work life
• Users judgment of the acceptability of the
  software

7
Analytic models of human performance with
computers

• 1980-1983 Card, Moran and Newell - significant
  advance from modeling in cognitive psychology
• modeled together many of the processes
  contributing to the full cycle of perception to
  action
• described in enough detail the knowledge
  necessary to perform a task
• Enabled to generate predictions about human
  behavior in real, naturalistic tasks

8
Framework

• Two key components
• Model Human Processor (MHP)
• General characterization of the human
  information-processing system
• System architecture
• Quantitative parameters of component performance
• GOMS
• A way of describing what the user needs to know
  to perform computer-based tasks

9
GOMS

• A family of models
• Describes
• The knowledge necessary
• Four cognitive components of skilled performance
  in tasks
• Goals
• Operators
• Methods
• Selection rules

10
Original GOMS Framework

• Focus
• Selection of
• methods from memory
• Time to specify
• and execute an action

11
The GOMS Method

• Assumptions
• Skilled user
• Serial sequence of independent cognitive
  operations and motor activities
• The method
• Predict a time it takes a user to execute a task
• A task is based on retrieving plans from
  long-term memory
• A method is chosen from available methods
  depending on the features of the task
• Execute motor movements necessary
• Time parameters for external actions were
  estimated from empirical data

12
Example

• Parameters
• k keystroke 280 msec
• M - mental operator 1.35 sec
• P pointing 1.1 sec
• H moving hands 400 msec
• Example
• sum ( a , b )
• Mkkk MkMkMkMkMk
• Total 6Ms 8 ks 6(1.35) 8(.280) 10.34
  sec

13
Limitations of GOMS

• Limited range of domains
• Applied to skilled users only
• Accounts for performance but neither learning nor
  recall
• Focused on errorless performance
• Gives little account of cognitive processes
• Focused of sequential tasks while many processes
  occur in parallel
• Does not address mental workload
• Disregards fatigue that users experience
• Does not account of individual differences among
  users

14
Advances in modeling specific serial components

• 1983 further research based on GOMS methodology
• Serial processing
• Time parameters are constant across tasks
• Incorporated relevant cognitive psychology
  factors
• Empirical work based on studies of entering
  editor commands, formulas in spreadsheets, etc.

15
Classes of parameters

• motor movement
• perception
• memory
• cognition

16
Motor Movements

• Keying
• Moving a mouse
• Hand movement

17
Keying

• Time to enter a keystroke in a normal typing task
• Value depends on
• The skill level of the typist
• Frequency with which a key is used
• Predictability and continuity of the text
• Example
• Skilled typist 80msec/keystroke
• User unfamiliar with the keyboard 1200 msec
  /keystroke

18
Moving a Mouse

• Pointing with a mouse at objects at various
  distances and of various target sizes.
• Derived from empirical experiments.
• Fittss law
• T 1.03.096 log2(D/S.5)
• applied to nested menus
• T .81.21 log2(D/S.5)

19
Hand Movements

• Time needed to move from the space bar of the
  keyboard until the pointing control begins to
  move the cursor.
• Large-muscle movement
• Characterized by Fittss law
• Empirically, T 360 msec

20
Perception

• Recognition of features of the current task and
  assessment of some parameters necessary to do a
  task
• Examples
• Time to respond to a brief light 100 msec
• (50-200 msec depending on intensity)
• Time to recognize a 6-letter word 340 msec
• Time for the eye to jump to next location
• 320 msec

21
Memory and Cognitive Processes

• Memory retrieval
• Executing steps in a mental procedure
• Choosing among methods

22
Memory Retrieval

• Time to retrieve the next unit of information
• well-known units
• from long-term memory to working memory
• A repeated act speeds up memory access

23
Memory Retrieval Example
Retrieve a command name or delimiter 1350msec
Retrieve a random command abbreviation 1200msec
Retrieve the next part of a formula 1100msec
Repeated retrieval of same command 660msec
24
Executing Steps in a Task

• GOMS catalogues
• the retrieval of goal and its subgoals
• the decision to select a method
• the retrieval of the motor movements
• the execution of each command component
• Production system formalism - explicit
  representation

25
Choosing Among Methods

• The more choices the longer the expected
  response time
• Empirical estimations of time vary
• (1.3 4.6 sec)

26
Composite Performance

• A task
• enter a block of values(2 digits)
• Mouse method- enter each value, point to the next
  cell with a mouse
• Menu method - ltretgt key advances cursor
  automatically to the next cell. Use mouse only to
  go to the next line

27
Empirical solution

• Empirical results
• Mouse method
• 4.19 sec per cell
• Menu method
• 2.46 sec per cell
• 2.81 sec to start each line

28
GOMS solution

• Mouse method
• moving the hand to the mouse 360 msec
• clicking the mouse 230 msec
• moving the hand to the keyboard 360 msec
• retrieving digits 1200 msec
• typing digits 460 msec
• retrieving the end action 1200 msec
• typing the ltretgt key 230 msec
• Total 4040 msec
• 3 error of 4.19 sec empirical result

29
GOMS solution (cont.)

• Menu method starting a new line
• moving hand to mouse 360 msec
• pointing to a new line 1500 msec
• clicking the mouse 230 msec
• moving hand to keyboard 360 msec
• Total 2450 msec
• 13 error of 2.81sec empirical result

30
GOMS solution (cont.)

• Menu method typing a number into a cell
• retrieving two digits 1200 msec
• typing two digits 460 msec
• retrieving the end action 1200 msec
• typing the ltretgt 230 msec
• Total 3090 msec
• 26 error of 2.46 sec empirical result

31
Pros and cons of the method

• Challenged based on inclusion or exclusion or an
  operation (esp. mental)
• Achievements
• within an average of 14 error of the observed
  values
• accurate enough to be useful

32
Summary

• Problems with GOMS
• Serial process assumption
• Independent task assumption
• Served well in a variety of basic computer-based
  tasks

33
Extension of the Basic Framework

• Learning and Transfer
• Time to learn
• Transfer from one system to the other
• Analysis of errors Workload in Working Memory
• Parallel Processes
• Modeling parallel processes with critical path
  analysis

34
Modeling in extended work

• Modeling of grammatical rules
• What knowledge a user must have before
  translating from goals to actions in a system?
• Similar to goal decomposition and methods in
  GOMS.
• Provide a countable entity
• The number of rules

35
Task-Action Grammar (TAG)
Feature Possible values
Direction Forward, backward
Unit Character, word
Taskdirection, unit SymbolDirection Letter Unit
Symbolforward cntl
Symbolbackward meta
Letterword W
Lettercharacter C
36
Task-Action Grammar (TAG)
Goals Action
Move cursor one character forward Cntl-C
Move cursor one character backward Meta-C
Move cursor one word forward Cntl-W
Move cursor one word backward Meta-W
37
Production system

• Make underlying knowledge explicit
• Once written, the accuracy and completeness can
  be checked by running the program
• Program can be used to predict both errors and
  learning time behavior

38
Rules for writing a SQL join query

• Rule 1 StartUp. SeeIfJoinNeeded
• IF
• GOAL SeeIfJoinNeeded
• Not (NOTE SeeingIfJoinNeeded TRUE
• THEN
• Add NOTE SeeingIfJoinNeeded TRUE
• Add STEP CountTables

39
Rules for writing a SQL join query

• Rule 2 CountTables
• IF
• GOAL SeeIfJoinNeeded
• STEP CountTables
• THEN
• Do Task Count NumberOfTables NumberOfTables
• Add NOTE NumberOfTables NumberOfTables
• Delete STEP CountTables
• Add STEP AddJoinNote

40
Rules for writing a SQL join query

• Rule 4 IfNumberOfTablesNot 2, ThenCleanUp
• IF
• GOAL SeeIfJoinNeeded
• STEP AddJoinNote
• NOTE NumberOfTables 1
• THEN
• Delete SETP AddJoinNote
• Delete NOTE NumberOfTables ?NumberOfTables
• Add STEP Cleanup

41
Use of Production Rules

• IF part
• Check for a match between the rules condition
  and the current goal and the current notes in
  working memory (WM).
• THEN part
• If there is a match, execute THEN part action to
  add and delete NOTES and STEPS to or from WM.

42
Time to Learn

• Kieras Polson
• Study learning under highly restrictive and
  controlled condition
• Determine the number of steps in a procedure
• The time to learn each step is 30 second
• The start-up time is 30 to 60 minutes.

43
Time to Learn (cont)

• Results from from widely different situations and
  labs are at the same order of magnitude.
• The number of rules is less critical than whether
  the features of those rules follow real-world
  features encoded in users memory.
• Problem how to quantify the learning time in
  more naturalistic situations.

44
Transfer of Training

• Production is the unit of learning.
• The number of productions shared by the two
  systems can predict the amount of transfer.
• Time to master a new procedure is a function of
  the number of new production to be learned.
• Specify the exact effects of consistent design
  across system and assess the relative costs of
  different degrees of consistency among procedures.

45
Transfer Predicted by the number of new rules to
be learned
46
Analysis of Errors

• Cause of errors
• Working Memory (WM) overload
• Production systems are used to estimate the
  contents of WM and the resident duration of each
  piece of information in WM.
• The more items in WM, the greater the likelihood
  of errors.

47
Systems with different WM workload

• Lotus 1-2-3
• User has to find and remember the coordinates of
  cells in the formula. (e.g. D23)
• Interactive Financial Planning System (IFPS)
• User can refer to a cells by name with adjective
  (e.g. previous) to indicate relative location. No
  need to remember the coordinates.

48
WM Load for Different Systems
49
Position Naming (e.g. B22) Formula with cells in
the same column

Col Name Row No.
Operator Var Name Var Name Var Name
Pointer Pointer Pointer Pointer Pointer Row No.
WM
Time
50
Keyword Naming (e.g. Previous Sales) Formula with
cells in the same column

Operator Var Name Var Name Var Name
Pointer Pointer Pointer Pointer Pointer Row No.
WM
Time
51
WM Load and Error Probability

• The higher the WM load, the more errors
• Lotus 19 items in WM, 14 errors
• IFPS 14 items in WM, 6 errors.
• When there are few than 8 items in WM, errors
  would be eliminated.
• Magic number 7

52
WM Load and Error Probability (cont)

• SQL join query
• the more the intervening restriction statements,
  the higher the probability of forgetting the last
  crucial join statement.
• No intervening restriction, 1.7 errors
• 2 intervening statements, 4.2 errors.

53
Problems with Error Analysis

• Do not distinguish between the peak load in WM
  and the duration of each item in WM.
• People might use strategy to reduce WM load.
  (Notes, clues from the environment)
• WM overload is not the only cause of errors.

54
Parallel Processes

• Reasons a serial model
• Easy to quantify
• task time sum of subcomponent time
• Tradition in Cognitive modeling
• Reasons for a parallel model
• External signals appear in parallel.
• Mental events can occur in parallel.
• External actions can be done in parallel.

55
Examples of Parallel Processes

• Rapid typing.
• Bank clerk enter the handwritten amount on the
  check into the computer.
• Menu search in expert users.
• The amount of parallel processes in a task
  depends on the skill level of the user.

56
Questions in Modeling Parallel Processes

• What processes are occurring in parallel?
• Which processes depend on each other so that they
  must occur in serial?
• The speed of which process is the limiting factor
  for the task?

57
Critical Path Analysis

• Analyze non-serial processes in timed cognitive
  tasks.
• Model cascading mental and external process.
• Analyze how three parallel processor resources
  (perceptual, cognitive, and motor) work with
  sequential dependencies.

58
Critical Path Analysis (cont)

• Parameters needed in critical path analysis
• Component processes
• Duration of each component process
• Dependencies among the component processes
• Total time taken by the task can be determined
  once the above parameters are all known.

59
Critical Path Analysis for 2 Typists
Perceptual processor
Cognitive
Motor processor
60
Critical Path Analysis for Different Users
61
What cognitive modeling in HCI can and can not do

• What cognitive model in HCI can do but more
  research is needed
• Non skilled or casual users
• Learning
• Errors and mental workload
• Cognitive processes
• Parallel processes
• Individual difference

62
What cognitive modeling in HCI can and can not
do (cont)

• What cognitive modeling in HCI can not do
• Estimate the impact of fatigue on performance
• Assess peoples acceptance to the interface
• Divide functionality between computer and human
  what human do best, what computer do best.
• How computer changes work and organizational life.

63
topic studied Extension possible insufficient information Different model
Non skilled user X
Learning X X
Errors X X
Cognitive Process X X X
Parallel Process X X X
Mental workload X
Functionality X
Fatigue X
Individual difference X
Acceptance X
Organizational life X