User Studies II

1
User Studies II

• With your instructor, Jeremy Hyland

2
Plan for Today

• Discuss the reading
• Why Johnny Cant Encrypt
• Johnny 2 Judgment Day
• Do a little testing of our own

3
Why Johnny Cant Encrypt

• Whos Johnny and why cant he encrypt?

Posner says
Whats Johnny trying to hide?
4
Why Johnny Cant Encrypt

• Whitten and Tygar, 1999
• http//www.usenix.org/publications/library/proceed
  ings/sec99/full_papers/whitten/whitten_html/index.
  html
• A Usability Evaluation of PGP 5.0

5
Why Johnny Cant Encrypt

• Security mechanisms are only effective when used
  correctly

So
If Usable then
else
6
Why Johnny Cant Encrypt

• Defining Usable Security Software
• Whitten and Tygar
• Security software is usable if the people who are
  expected to use it
• are reliably made aware of the security tasks
  they need to perform.
• are able to figure out how to successfully
  perform those tasks
• don't make dangerous errors
• are sufficiently comfortable with the interface
  to continue using it.

7
Why Johnny Cant Encrypt

• Why is usable security hard?

McNealy says
You have no usable security, get over it.
8
Why Johnny Cant Encrypt

• Why is usable security hard?
• Five reasons
• 1. The unmotivated users
• Security is usually a secondary goal
• 2. Policy Abstraction
• Programmers understand the representation but
  normal users have no background knowledge.

9
Why Johnny Cant Encrypt

• Why is usable security hard?
• Five reasons
• 3. The lack of feedback
• We cant predict every situation.
• 4. The proverbial barn door
• Need to focus on error prevention.
• 5. The weakest link
• Attacker only needs to find one vulnerability

10
Why Johnny Cant Encrypt

• Usability Evaluation
• PGP 5.0
• Pretty Good Privacy
• Software for encrypting and signing data
• Plug-in provides easy use with email clients
• Modern GUI, well designed by most standards

11
Why Johnny Cant Encrypt

• Usability Evaluation
• Whitten and Tygar focus their evaluation on a
  question based off their definition of usable
  secure software
• If an average user of email feels the need for
  privacy and authentication, and acquires PGP with
  that purpose in mind, will PGP's current design
  allow that person to realize what needs to be
  done, figure out how to do it, and avoid
  dangerous errors, without becoming so frustrated
  that he or she decides to give up on using PGP
  after all?

Loaded question?
12
Why Johnny Cant Encrypt

• Usability Evaluation
• Cognitive walk through
• Mentally step through the software as if we were
  a new user. Attempt to identify the usability
  pitfalls.
• Focus on interface learnablity.

13
Why Johnny Cant Encrypt

• Usability Evaluation
• Cognitive walk through results
• Visual metaphors
• Public vs. Private keys
• Signatures and verification

14
Why Johnny Cant Encrypt

• Usability Evaluation
• Cognitive walk through results
• Different key types
• Compatibility increases complexity
• Keys listed as users

15
Why Johnny Cant Encrypt
Keys listed as users
16
Why Johnny Cant Encrypt

• Usability Evaluation
• Cognitive walk through results
• Key server
• Hidden?
• What is it doing?
• Revocation not automatic

Would that help?
17
Why Johnny Cant Encrypt

• Usability Evaluation
• Cognitive walk through results
• Key management policy
• Unneeded confusion
• Whats the difference between trust and validity?

18
Why Johnny Cant Encrypt

• Usability Evaluation
• Cognitive walk through results
• Irreversible actions
• Need to prevent costly errors
• Consistency
• Encoding?!?
• Too much information
• More unneeded confusion
• Show the basic information, make more advanced
  information available only when needed.

19
Why Johnny Cant Encrypt

• Usability Evaluation
• User Test
• PGP 5.0 with Eudora
• 12 participants all with at least some college
  and none with advanced knowledge of encryption
• Participants were given a scenario with tasks to
  complete within 90 min
• Tasks built on each other
• Participants could ask some questions through
  email

20
Why Johnny Cant Encrypt

• Usability Evaluation
• User Test Results
• 3 users accidentally sent the message in clear
  text
• 7 users used their public key to encrypt and only
  2 of the 7 figured out how to correct the problem
• Only 2 users were able to decrypt without
  problems
• Only 1 user figured out how to deal with RSA keys
  correctly.
• A total of 3 users were able to successfully
  complete the basic process of sending and
  receiving encrypted emails.
• One user was not able to encrypt at all

21
Why Johnny Cant Encrypt

• Conclusion
• If an average user of email feels the need for
  privacy and authentication, and acquires PGP with
  that purpose in mind, will PGP's current design
  allow that person to realize what needs to be
  done, figure out how to do it, and avoid
  dangerous errors, without becoming so frustrated
  that he or she decides to give up on using PGP
  after all?
• Nope
• Is this a failure in the design of the PGP 5.0
  interface or is it a function of the problem of
  traditional usable design vs. design for usable
  secure systems?
• Security as the primary function vs. a secondary
  function

22
Johnny 2

• Garfinkel and Miller, 2005
• http//www.simson.net/clips/academic/2005.SOUPS.jo
  hnny2.pdf
• Follow-up to Why Johnny Cant encrypt
• Test of new encryption technology
• Key Continuity Management
• S/MIME certificates
• Better interface
• Simple buttons

23
Johnny 2

• Garfinkel and Miller
• Johnny couldnt encrypt because of the key
  architecture behind PGP.
• .the fundamental usability barriers that
  Whitten identified could be overcome by replacing
  the underlying third-party certification model
  with Key Continuity Management.

24
Johnny 2

• User Test
• Tried to stay as close to the Johnny experiment
  as practical
• Same methods of user solicitation/selection
• Same basic scenario
• Similar user tasks
• Added attackers

25
Johnny 2

• User Test
• Attacks
• new key attack
• new identity attack
• unsigned message attack
• How well does the interface enable users to
  respond to these attacks?

26
Johnny 2

• User Test
• Test application CoPilot
• Wizard of Oz prototype
• S/MIME certificate handling
• First time Yellow
• Trusted certificate Green
• Changed certificate Red
• Unsigned message White
• Unsigned message from a sender that normal sends
  signed messages Gray
• Better tools allow for a more automated and
  scientific test

27
Johnny 2

• User Test
• 43 test subjects
• Three groups
• No KCM
• Color
• ColorBriefing

28
Johnny 2

• User Test
• Results
• Users generally understood the basics
• Little understanding of signature integrity
  guarantees
• Verifying attack message authenticity was
  difficult for most users
• No group resisted attacks 100 of the time
• Color and ColorBriefing resisted new key attack
  and the unsigned message attack better then No
  KCM
• The interface did not help against new identity
  attacks

29
Johnny 2

• User Test Conclusions
• A few surface interface issues
• Do not trust button
• Misconceptions about the security of sealed
  messages
• Generally, the new interface simplifies email
  encryption
• Still problems with determining certificate
  trust, however some of these problems may be
  unavoidable.

30
So Now What?

• Now its time to do your own test!

31
User Test

• 3 groups
• Cell Phone
• CD player
• Calculator
• Take a few minutes to create a simple user test
• One member of each group switches to be a tester

32
User Test

• Guidance
• Decide whose going to do what!
• Create a Use Case Scenario
• Define user tasks for completion of the scenario
• Set up metrics for results evaluation
• What qualifies as success vs. failure?

33
User Test

• Results!?