NIST SAMATE Project and OMG

1
NIST SAMATE Project and OMG

• Michael Kass
• NIST Information Technology Laboratory
• http//samate.nist.gov
• March 11, 2008

2
Overview

• NIST SAMATE Project
• Testing the Tools
• Automated Test Case Generation
• CWE Formalization
• SAMATE and CWE Effectiveness Program
• TCG Where are we now?
• Other SAMATE work

3
SAMATESoftware Assurance Metrics and Tool
Evaluation Project

• Co-sponsored by DHS to
• Create tests and tool specifications for software
  assurance (SwA) tool evaluations
• Develop metrics for measuring SwA tool
  effectiveness
• Identify gaps in current SwA technology
• Make recommendations to DHS for areas of research

4
Testing the Tools

• SAMATE Reference Dataset (SRD)
• Online repository of tool tests
• Thousands of source code samples containing
  examples of CWEs
• Discrete tests developed by NIST, contributed
  by tool developers, academia and public
• Tests are based upon interpretation of a
  particular weakness definition (currently no
  formal white-box definitions)
• Tests are freely available at http//samate.nist.g
  ov/SRD

5
Automated Test Case Generation (TCG)
Formal CWE Definitions (SBVR/KDM)
KDM
Tool Tests (code)
Code Analysis Tool

• Funded by DHS
• Part of SAMATE effort to expand SRD to cover as
  many CWEs as possible
• Based upon OMG MDA Technology (MOF, UML, XMI)
• Uses formalized CWE definitions (SBVR)
• Contractual Formalization that is based on OMG
  standard, Semantics of Business Vocabulary and
  Rules (SBVR) and
• Technical Formalization that is based on OMG
  standard, Knowledge Discovery Metamodel (KDM)

6
CWE Formalization

• White Box Definitions Focus on the structure
  patterns of the inner components and their
  interactions (that determine certain observable
  behavior)
• Provide compliance points that
• Describe patterns of code (as they can be
  directly identified in code)
• Identify discernable properties of patterns of
  code
• Enable automation
• Enable direct step-by-step comparisons of the
  decision procedures implemented within tool

7
SAMATE and CWE Effectiveness Program

• Long-term goal To auto-generate tool tests
  using formal CWE definitions in collaboration
  with MITREs CWE Effectiveness program
• Provide tests ad hoc to tool developers
• Developers run tests against their tool
• Developers can publish test results

8
TCG Where are we now?

• TCG Status
• Can generate tests for 3 CWEs
• Near term, NIST will expand formal CWE
  definitions to 25 high priority CWEs based
  upon their
• Occurrence
• Severity
• Recognized by tools today
• Long term, TGC will cover as many CWEs as
  possible
• With coding complexities

9
Other SAMATE Projects

• Ongoing work
• Developing tests for web application scanners
• Adding to existing tests for source code security
  analyzers
• Performing tool effectiveness studies
• New areas
• Testing binary analyzers
• The static analyzer tool exposition (SATE)
• Software transparency/pedigree information
• Malware research protocols