State Machine Architecture Validation Toolset

1
State Machine Architecture Validation Toolset

• Kevin Gary
• Division of Computing Studies
• Arizona State University

2
IGSTK Architecture

• Mission-critical applications require safety!
• Safety achieved by
• Component-based safety-by-design
• Continuous and complete testing
• Software process best practices
• Components driven by State Machines

3
State Machines

• Encapsulated within each component
• No visibility or external manipulation
• Govern component responses to requests
• Coded by hand

IGSTK Component
State Machineltgt
public
RequestAction1()
RequestAction2()
private
PerformAction1()
PerformAction2()
4
State Machine Validation

• Risk State machines hand-coded, as is the
  underlying runtime semantics
• Solution Tools that validate SMs
• Structurally sound - static analysis
• Verification - behave as expected/predicted
• Validity - always in a valid state
• Consistency - navigated states and transitions
  consistent with application behavior
• Constraints on states and transitions visited

5
IGSTK Validation Toolset

• Started September 2005
• Tested with IGSTK Iteration 8 Release
• Integrating with DART/CMake
• Current Capabilities
• Replay IGSTK execution
• SM Coverage tools
• Visualization/Animation
• Dependencies
• Java, C, Ant
• SCXML, LTSA

6
Current Toolset Capabilities

• Replay - Parses IGSTK logfile and extracts SM
  inputs to a reconstructed event stream
• Developers can verify expectations
• Troubleshooting
• Replays can be animated (stay tuned)
• Visualization Animation - Maps output of a
  replay or test case to LTSA animation (scenebean)
  format.
• LTSA compliments of Magee Kramer
• Next steps rewrite as native C to avoid
  license issues

7
Current Toolset Capabilities

• State Machine Coverage
• IGSTK goal 100 unit test code coverage
• SM Validation all states/transitions verified
• Node coverage - generate event sequence that
  guarantees every node is visited.
• Edge coverage - same for every transition
• Current work
• Integrating into DART dashboard
• Heuristic path coverage
• Next steps
• Consider complexity of n components and m
  processors

8
Proposal Tasks

• Enhance the current toolset to support validation
  in concurrent scenarios
• Provide support for large-scale simulation-based
  testing
• Add the capability to validate global system
  state
• Provide facilities to make IGSTK easier to use
  for application and component developers
• These are discussed in reverse order next

9
Task 4 Ease of Use

• Follow work on application helper tool
• If state machines are recommended to application
  developers, then ensure they can easily use the
  validation tools to assist them in designing and
  troubleshooting issues early.
• In other words, more than just scriptable for
  DART.
• Write a new visualizer tool
• LTSA is nice, but there are some issues
• Not available for BSD-style open source license
• In Java (not that big a deal)
• Would like to do some functional upgrades

10
Task 3 Validate global state

• IGSTK considers SMs on a per-component basis
• By design! This ensures encapsulation
• The state of one object is not dependent on the
  state of another object (loosely coupled)
• But, the state of one component might need to be
  consistent with another component (cohesion)
• IGSTKs layered architecture implies this
• Checking Global System State
• System state is the union of states of the
  components
• We need to ensure the union is consistent
• Rule-checking specify and evaluate predicates
  that express constraints on globally valid states

11
Task 2 Large-scale simulation

• Path coverage is intractable
• Perhaps doable on a per-component basis, but over
  the global state of an IGSTK application?
• Additionally, there are some translation steps
  from real values to input types - so the possible
  values (or errors) in these values may matter.
• Simulation
• Use heuristic techniques to do the best job as
  possible with path coverage
• Add Mock Objects to our current test-case
  generation capabilities to simulate what happens
  on the owner object
• Can create a realistic distribution of errors
• Envision a test-vase generation tool where
  particular mocks are tuned to focus tests
  (happy day case, negative tests)

12
Task 1 MT and IP communication

• This means concurrency - a complex problem from a
  validation perspective
• Deadlock, starvation, race conditions
• Difficult for humans to completely identify in
  analysis
• First two may not be too bad as our components
  are pretty independent from a resource
  perspective (I think?)
• Still worry about race conditions causing dirty
  reads, lost writes, or cascading failed requests
• May wish to look at formal methods such as model
  checkers to try and detect in our designs.
• State space explosion
• Particularly if multiple processors for multiple
  threads
• Proxies would add additional request initiators
• Our node/edge/path coverage has to be revisited

13
Summary

• What we are currently looking at
• Coverage algorithms
• Replay and simulation
• New visualizer
• Better DART/Cmake integration into toolkit
• What we want to look at next
• Rule checking for dynamic global state checks
• Mock objects to improve simulation
• Concurrency - validation problem depends on
  design so we will follow these discussions
  closely
• Students Shylaja Kokoori (on leave), Srinivas
  Busam, Benjamin Muffih, others