Usability and Accessibility (U

1
Usability and Accessibility (UA) Research Update

• Sharon J. Laskowski, Ph.D.
• http//vote.nist.gov

2
Outline

• UA research goals
• Overview of all current research efforts
• Test method validation
• Tester qualifications, test documentation and
  workflow
• New test methods
• Improving standards
• Specific details
• Validation of Voter Performance Protocol
• Tester qualifications, test documentation and
  workflow
• Accessibility research

3
UA Research Goals

• Improving test methods
• Test validation
• Tester qualifications, test documentation and
  workflow
• New test methods
• Improving standards
• Accessibility is the current focus
• See 2010 UA Working Group paper on VVSG 2.0 UA
  issues and gaps

4
Overview of Current Research Efforts

• Test method validation
• Voter Performance Protocol
• The usability performance benchmarks test from
  VVSG 2.0
• UA design inspection tests
• Accessibility throughout the voting session
• The end-to-end accessibility test
• Usability for poll workers test

5
Overview (Contd)

• Tester qualifications, test documentation and
  workflow
• New test methods
• Accessibility performance test
• Improving standards
• Dexterity research
• Audio-tactile interface research

6
Validation of Voter Performance Protocol (VPP)

• Goal Assess the reproducibility of the VPP
  performance benchmark test in different
  geographic locations
• Data collection
• DRE and optical scanner
• 5 locations
• 125 voters/location
• Straight party vs. no straight party voting states

7
VPP (Contd)

• Locations
• DC Metro area assessing reproducibility of
  earlier DC study
• Texas Indiana states with straight party
  voting
• Colorado Tennessee states w/o straight party
• Data analysis now under way
• Tennessee more difficulties with completion on
  DREs
• Indiana issues with recruiting to the specified
  test participant demographics

8
UA Tester Qualifications

• Voting System Test Labs (VSTLs) need to select
  contractors who are qualified to execute UA
  tests of the VVSG requirements
• National Voluntary Laboratory Accreditation
  Program (NVLAP) needs to assess UA proficiency
  and skills for the accreditation of VSTLs
• We have developed assessments
• Based on our discussions with testers involved in
  our validation efforts and with the VSTLs

9
Assessment and Training for Testers

• Assessment factors we have developed a
  checklist
• UA expertise and experience, including testing
  participants with specific disabilities and
  assistive technology
• Skills in planning and performing tests
• Voting system domain knowledge, VVSG familiarity
• Most candidates will have strong skills on UA
  testing, but weak domain knowledge
• VSTLs would be responsible for training their
  contractors on voting systems, testing to
  standards, etc.

10
Proficiency Test Questions

• NVLAP lab accreditation should include evaluating
  VSTL UA testers proficiency in areas like
• Voting system domain knowledge
• Familiarity with VVSG and test methods for UA
• We have developed specific questions and a
  grading rubric, for example
• How would you test 3.2.2.c The voting system
  shall provide the voter the opportunity to
  correct the ballot for either an undervote or
  overvote before the ballot is cast and counted.
  ?
• What do you do with the manufacturer's summative
  usability testing report in the TDP?
• Whats the difference between design inspections
  and performance tests?

11
Test Method Documentation

• Tester qualifications are not sufficient test
  method documentation plays a significant role in
  tester performance
• Documentation usability form and content
• Plain language
• Document navigation
• Multi-platform
• Cookbook quality all supporting material
  together in a single package
• Scripts, screeners, protocols, NISTIRs, etc.
• Include ancillary sources along with the test
  methods
• Package for each major test method

12
Workflows for Test Method Execution

• Organize workflow by number of
• Election rebuilds, testers, test participants
• Weigh multiple factors in designing workflow for
  efficiency
• Machine class and capabilities
• Ballot styles needed for a given test
• Coordination with other test teams, such as
  software or security, when conducting tests
• Need more information from VSTLs on how they
  conduct test campaigns

13
Development and Management of Test Methods

• In general
• Need a feedback loop with VSTLs, EAC, NIST,
  NVLAP, election officials
• Lifecycle management of tests and all associated
  documentation
• Living standards process
• Need coordination across organizations

14
Accessibility Research

• NIST research supports incremental improvements
  to voting system standards
• Focus is on dexterity and audio-tactile interface
  research
• Examining research, best practices and standards
  in other domains and exploring how these apply to
  voting systems
• Performing small research studies to verify
  applicability
• EAC Accessible Voting Technology Initiative
• NIST providing technical support, especially for
  current state of the art, testing and evaluation

15
Discussion/Questions
Page 15