The Cognitive Walkthrough

1
The Cognitive Walkthrough
andCognitive Walkthrough for the Web--A
Worked Example
(Computer Mediated Communication) (René van der
Ark) (RuG)
2
The Cognitive Walkthrough

• From
• Testing a Walkthrough Methodology for
  Theory-Based Design of Walk-up-and-Use
  Interfaces, Lewis, Polson, Et al.

3
The Cognitive Walkthrough Background
03/40

• Based on a theory of exploratory learning
• CE model (Polson Lewis)
• Results in series of theoretically motivated
  questions for evaluation of a user interface
• Is used with applications with minimal
  training-requirements

4
The Cognitive WalkthroughFormat of presentation
04/40

• 1. CE model superficial explanation
• 2. Guidelines derived from CE
• 3. Details of the Cognitive Walkthrough
• 4. Evaluation of the method

5
The Cognitive Walkthrough 1. The CE Model
05/40

• The CE Model for Exploratory Learning
• 3 components
• Problem solving component
• Learning component
• Execution component

6
The Cognitive Walkthrough 1. The CE Model
06/40

• Problem Solving phase
• Action choice of user
• - based on similarity between his/her
  expectation of actions consequence and his/her
  goal
• Cause for choice
• - Match beween description of action and goal can
  cause user to choose this action
• Response Evaluation
• User seeks match between goal and computer
  response evaluation
• A mismatch results in an attempt to undo the
  action

7
The Cognitive Walkthrough1. The CE Model
07/40

• Learning Phase
• Learning occurs when
• Evaluation leads to a positive decision
• The Problem-Solving step is stored in users
  memory as a new rule

8
The Cognitive Walkthrough1. The CE Model
08/40

• Learning Phase
• Major problems in learning
• Due to difficulty complexity of
  problem-solving process
• Not due to encoding processes that store
  succesful problem-solving episodes in long-term
  memory
• i.e. responsibility moves from user to designer!

9
The Cognitive Walkthrough1. The CE Model
09/40

• Execution phase
• Users first fire rules to find a rule
  applicable to the current context
• If no applicable rule is found the
  problem-solving phase is invoked

10
The Cognitive Walkthrough2. Design for
Successful Guessing
10/40

• Lewis Polson Knowledge-poor problem-solving
  strategies () are a guessing process - CE
• Hence UI-Design for Succesful Guessing

11
The Cognitive Walkthrough2. Design for
Successful Guessing
11/40

• Four Most important guidelines (1/2)
• Make the reportory of availabe actions salient
• (user should understand all given options)
• (user must be able to reach all given options)
• Provide an obvious way to undo actions
• (user must be allowed to make mistakes in order
  to learn)

12
The Cognitive Walkthrough2. Design for
Successful Guessing
12/40

• Four Most important guidelines (3/4)
• Offer few alternatives
• Require as few choices as possible
• Conflict 3 and 4
• This implies use of both a narrow and a deep
  menu-structure!
• Solution
• If a choice is clear (semantically) user can
  distinguish right choice from 10-15 options

13
The Cognitive Walkthrough3. Details of the
Cognitive Walkthrough
13/40

• The Cognitive Walkthrough is
• A set of questions intended to focus the
  designers attention on problem-solving- and
  learning processes

14
The Cognitive Walkthrough3. Details of the
Cognitive Walkthrough
14/40

• The process
• A The designer specifies a series of
  action-tasks to evaluate
• B The designer specifies steps to perform for
  succes in the task
• C Each step is evaluated

15
The Cognitive Walkthrough3. Details of the
Cognitive Walkthrough
15/40

• Evaluation Step 1 (Q.1 Q.2)
• Evaluator specifies
• Users current goal
• The next action the user should take

16
The Cognitive Walkthrough3. Details of the
Cognitive Walkthrough
16/40

• Evaluation Step 2 (Q.2a-Q.7)
• Evaluator judges the ease with which
• The user is able to correctly select an action
• The user is able to correctly execute the action

17
The Cognitive Walkthrough3. Details of the
Cognitive Walkthrough
17/40

• Evaluation Step 3 (Q.8)
• Evaluator evaluates
• System Response
• Adequacy of System Response

18
The Cognitive Walkthrough3. Details of the
Cognitive Walkthrough
18/40

• Evaluation Step 4 (Q.9)
• Evaluator evaluates
• Can the user form an appropriate next goal?
• in this case go back to step 1
• OR
• Is the task successfully completed?

19
The Cognitive Walkthrough4. Evaluation of the
Cognitive Walkthrough
19/40

• Advantages
• Explicitates important implicit design decisions
• Theory Testing are combined ad hoc
• Detailed understanding of problem solving and
  learning components

20
The Cognitive Walkthrough4. Evaluation of the
Cognitive Walkthrough
20/40

• Disadvantages
• Using theoretical model can lead to conflicting
  guidelines
• A complete thorough analysis is time consuming

21
The Cognitive Walkthrough4. Evaluation of the
Cognitive Walkthrough
21/40

• Effectiveness of the method
• Issues before evaluating the method
• Would the technique give consistent results?
• Would the technique come to the same conclusions
  as empirically acquired usability data (of the
  tested UIs)?

22
The Cognitive Walkthrough4. Evaluation of the
Cognitive Walkthrough
22/40

• Effectiveness of the method
• Four different UI designs studied
• CW predicted 70 out of 124 action-paths
  (traversals) the users took in emperical studies
• CW predicted 51/105 traversals leading to errors
• CW detects approx. 50 of the problems revealed
  by extensive empirical evaluation

23
The Cognitive Walkthrough4. Evaluation of the
Cognitive Walkthrough
23/40

• Final Note
• Inconsistency between evaluators
• 3 Evaluators with intimite knowledge of theory
  predicted more traversals than
• 1 Evaluator without intimite knowledge
• Concluding
• Cognitive Walkthrough requires expert knowledge
  of cognitive learning theory

24
Cognitive Walkthrough for the Web--A Worked
Example

• From
• Cognitive Walkthrough for the Web, Blackmon,
  Polson, Et al.
• and
• A solution to Platos Problem The latent
  Semantic Analysis Theory of Acquisition,
  Induction and Representation of Knowledge,
  Landauer, Dumais

25
Cognitive Walkthrough for the Web
25/40

• Cognitive Walkthrough for the Web (CWW) features
• Contextually rich descriptions of user goals
• Iteration into subsequent sub-pages
• Different organisation suitable for the web

26
Cognitive Walkthrough for the Web
26/40

• The Comlpete Procedure
• Detailed description of the website
• Rough outline of successor-pages
• Iterative process through successor-pages

27
Cognitive Walkthrough for the WebCWW as an
extension to CW
27/40

• CW
• Q1 Will the correct action be made sufficiently
  evident to the user?
• Q2 Will the user connect the correct actions
  description with what he/she is trying to do?
• Q3 Will the user interpret the systems response
  to the chosen action correctly?

28
Cognitive Walkthrough for the WebCWW as an
extension to CW
28/40

• CW
• Q2 Will the user connect the correct actions
  description with what he/she is trying to do?

• CWW
• Q2a Will the user connect the correct subregion
  of the page with the goal using heading
  information and his/her understanding of the
  sites page-layout conventions?

29
Cognitive Walkthrough for the WebCWW as an
extension to CW
29/40

• CW
• Q2 Will the user connect the correct actions
  description with what he/she is trying to do?

• CWW
• Q2b Will the user connect the goal with the
  correct widget in the attended subregion of the
  page using link-labels and other kinds of
  descriptive information?

30
Cognitive Walkthrough for the WebBackground
30/40

• CWW based on CoLiDeS
• Comprehension-based Linked Model of Deliberate
  Search (Kitajima, Blackmon, Polson)
• Consensus information scent drive users
  information seeking behavior.
• User chooses option most semantically similar to
  his/her current goal

31
Cognitive Walkthrough for the WebBackground
31/40

• CWW uses LSA
• Latent Semantic Analysis (Landauer, Dumais)
• Estimate semantic relatedness of texts using
  Information Retrieval-techniques
• LSA enables CWW to use narrative descriptions of
  user goals

32
Cognitive Walkthrough for the WebApplied to a
webpage
32/40

• 4-step analysis
• Step 1
• Compile set of realistic user goals (100-200
  words)
• Find the correct actions to take on the website
• Define the semantic space

33
Cognitive Walkthrough for the WebApplied to a
webpage
33/40

• 4-step analysis
• Step 2 (LSA)
• Compare user goals to availabe links/headings
• 1 to many comparison on goal-narrative and links
• Determine whether links are understandable
• Calculate vector lengths to semantic space
• Analyse link-coherence
• Matrix analysis comparing all available links
  with each other

34
Cognitive Walkthrough for the WebApplied to a
webpage
34/40

• 4-step analysis
• Step 3
• Look for unfamiliarity of the links using
  vector-lengths
• Vector length lt 0.8
• Look for confusable links.
• Coherence score gt 0.6

35
Cognitive Walkthrough for the WebApplied to a
webpage
35/40

• 4-step analysis
• Step 4
• Look for goal-specific competing links (3
  criteria)
• Competing link-label must be under the same
  heading as the correct link
• Must have a cosine score to the goal of at least
  80 of the score of the correct link
• Evaluator does not judge the link as a false alarm

36
Cognitive Walkthrough for the WebThe Worked
Example
36/40

• Scenario
• For a small research-paper, on the subject of
  CMC HCI, I was referred to an article on the
  web. I was told this should be easy to find
  through the RuG-website link to the ACM Digital
  library.

37
Cognitive Walkthrough for the WebThe Worked
Example
37/40

• Goals
• Iteration 1 "Find the section for the online
  article databases"
• Iteration 2 Find the section for articles on
  the web
• Etc.

38
Cognitive Walkthrough for the WebThe Worked
Example
38/40

• Correct actions
• Iteration 1 Select Library
• Iteration 2 Select Electronic Databases
• Etc.

39
Please wait

• We will now switch to the demonstration

40
References

• Testing a Walkthrough Methodology for
  Theory-Based Design of Walk-up-and-Use
  Interfaces, Lewis, Polson, Et al.
• Cognitive Walkthrough for the Web, Blackmon,
  Polson, Et al.
• A solution to Platos Problem The latent
  Semantic Analysis Theory of Acquisition,
  Induction and Representation of Knowledge,
  Landauer, Dumais
• Comprehension-based Model of Web Navigation and
  its Application to Web Usability Analysis,
  Blackmon, Polson, Et al.