Michael Kass

1

Update on SAMATE Automated Source Code
Conformance Verification Against VVSG 1.0
Requirements

• Michael Kass
• National Institute of Standards and Technology
• http//vote.nist.gov

2
Outline

• Progress report on automated source code
  conformance verification effort introduced at
  last TGDC meeting
• Lessons learned
• Next steps

3
Software Assurance Metrics and Tool Evaluation
(SAMATE)

• NIST/DHS co-sponsored project
• Goal - measure the effectiveness of software
  assurance tools to identify weaknesses and
  vulnerabilities in software
• Asked by EAC to assist voting system
  manufacturers and test labs improve a (heavily
  manual) source code review process by developing
  a uniform automated conformance verification
  capability for voting system source code against
  VVSG 1.0 software requirements

4
Tool Automation Progress

• Finished and tested automation of VVSG 1.0 rules
  for Java (still documenting C/C work)
• 50 custom rules written for the Checkstyle open
  source Java code analysis tool
• 41 verification tests (sample bad code) against
  those rules, to verify tool correctness against
  requirements
• Ran customized tool against 1M lines of Java
  source code
• Bundled Java tool rules and tests into ZIP and
  tar archive files for distribution to TGDC, VSTLs
  and manufacturers
• 1 "National Information Assurance Glossary"
  CNSS Instruction No. 4009 National Information
  Assurance Glossary

5
Requirement Automation Overview

• Identified 58 software requirements in VVSG
  (since last TGDC report)
• 26 requirements fully/partially automated for
  Java
• 12 requirements do not apply to Java language
• 14 requirements require entirely human analysis
• 3 requirements could be automated, but require
  custom knowledge about code or documentation
• 2 requirements could be verified with a more
  capable tool
• 1 requirement is ambiguous (not automatable)

6
Findings Requirement Interpretation Issues

• We identified 10 requirement interpretation
  issues that require clarification
• The term module is often used in VVSG 1.0
  without reference to type (method/function,
  class, library)
• What constitutes intentional exception is not
  defined
• Logical nesting limit requirement needs clearer
  definition
• Scoping clarification needed for names that
  differ by only one character
• Module line count requirement (type of module,
  and which lines count) not defined (we assumed
  modulemethod)

7
Findings Requirement Interpretation Issues

• Additional requirement clarification needed for
• Indentation requirements not specific (clear
  and consistent is all the requirement
  specifies)
• Scoping clarification needed for unique name
  comparison in code
• Functional, test and branch statements not
  clearly defined
• Definition of what a library module is for Java
  is not specified
• Lowest shared access for a resource is not
  clearly defined

8
Findings Even Clear Requirements May Be
Interpreted Differently

• An example Initializes every variable upon
  declaration where permitted (VVSG 1.0 25.4.2)
• int varOne0 // VVSG conformant
• static int varTwo // non-VVSG-conformant
• What if Java automatically initializes varTwo to
  a value of 0 for you? In some cases, this
  happens at runtime
• Some manufacturers believe that the VVSG
  requirement is satisfied without explicitly
  assigning a value to a variable at declaration,
  and have written their code that way

9
Findings Tool Automation Requires An Unambiguous
Specification

• Need mutual understanding of VVSG requirement
  meaning among voting system manufacturers, VSTLS
  and the EAC
• The devil is in the details to unambiguously
  specify requirements for source code
• We observed some disconnects between voting
  system code and SAMATE interpretation of VVSG
  requirements (from our July 2011 visit to Wyle
  Labs)

10
Findings Where Possible, SAMATE Should
Customize Tools Manufacturers are Already Using

• Where voting system manufacturers are already
  using automated tools, SAMATE should (first) try
  to customize the tools that manufacturers are
  using
• Two manufacturers currently using Checkstyle
  tool for Java code conformance verification
• SAMATE rules can complement the Checkstyle rules
  the manufacturer already uses without adding
  another tool
• This requires a dialog between SAMATE and
  manufacturers to identify which tools they are
  using

11
Findings Need to Maximize SAMATE Resources

• To make best use of our resources, SAMATE needs
  to know which tools, programming languages,
  compilers, libraries and coding conventions
  manufacturers are using
• 70 of voting system source code is written in
  C/C, Java and C languages (from discussion
  with VSTLs)
• Manufacturers compiler/library selection also
  dictate which tools SAMATE chooses for automation
  verification (e.g. GNU vs. Microsoft C)
• If manufacturers are using industry coding
  conventions (not VVSG 1.0 requirements), then
  SAMATE should focus its automated verification
  efforts against those industry coding conventions
• Synchronize SAMATE work with future VSTL testing
  work

12
Next Steps

• TGDC review of customized Java tool and tests
• Discuss any issues with the SAMATE work
• Resolve those issues
• Distribute SAMATE tool rules and tests to
  manufacturers and VSTLs
• Two manufacturers are using the Checkstyle tool
  now (using their own custom rules)
• Get manufacturer/VSTL feedback on their
  interpretation and implementation of VVSG
  requirements in their tools
• Resolve those differences (via RFI or other
  means)
• Incorporate changes into future releases of VVSG
  as appropriate
• Publish a final distribution of customized tools
  and tests

13
Next Steps

• Have a discussion with voting system
  manufacturers and VSTLs to determine where SAMATE
  should focus its future efforts in
• Tools
• Programming languages and compilers
• Coding conventions
• Future VSTL work (new systems, re-certifications)

14
Next Steps

• Investigate how SAMATE can contribute to
  automated conformance verification against VVSG
  1.1 software integrity requirements
• Most VVSG 1.0 style requirements are removed in
  VVSG 1.1 (in lieu of using industry-accepted
  coding style conventions)
• VVSG 1.1 requirements are more integrity
  focused
• Not trivial to find integrity bugs in source code
• Requires more sophisticated tools than style
  checkers
• Requirements include identifying race conditions,
  deadlocks, livelocks, pointer validation, dynamic
  memory allocation, numeric overflows and CPU
  traps

15
Next Steps

• Integrate SAMATE VVSG work with SAMATE Common
  Weakness Enumeration (CWE) Compatibility
  Effectiveness program for testing automated
  vulnerability analysis tools
• CWE is a unified, measurable set of software
  weaknesses defined in an online dictionary
  sponsored by DHS and implemented by MITRE
• Tools are assessed against their ability to
  identify CWEs in software with known weaknesses
  and known source code complexities (that can
  influence a tools ability to correctly identify
  a weakness)
• SAMATE can leverage this work in developing test
  suites to verify tool effectiveness against VVSG
  source code integrity requirements

16
Summary

• SAMATE was successful at fully or partially
  automating Java source code verification against
  VVSG 1.0 requirements that could be automated
• Need to resolve semantic definitions of some VVSG
  1.0 software requirements (with manufacturers,
  labs and EAC) for uniform automated conformance
  verification uniformly across all tools
• Need to prioritize SAMATE resources against
  voting system manufacturers tools, languages,
  compilers and coding conventions to maximize
  impact for future work
• Need to look forward to VVSG 1.1 software
  integrity requirements, and how SAMATE can assist
  manufacturers and VSTLs through tool
  effectiveness testing

17
Discussion