Standards Conformance Testing and M

1
Standards Conformance Testing and MS Web
Services

• Bernard P. Zeigler, Ph.D.,
• Arizona Center for Integrative Modeling and
  Simulation
• and
• Joint Interoperability Test Command
• Fort Huachuca, AZ 85613-7051

2
Outline

• Role of Standards Conformance Testing in
  Net-Centric TE
• JITC responsibility for Link-16 standards
  conformance testing
• Link-16 Challenges to implementation and testing
• MSbased Automated Test Case Generation
• Application to Link-16 in the IABM SIAP Context
• Service Oriented Architecture MS-based Testing
  Services
• Link-16 standards conformance testing
• certification for GIG deployable services
• exposing MS-based TE assets as services

3
Net-Centric Test and Evaluation Some Assertions

• Net-Centric TE
• places an increased emphasis on standards
  conformance testing (SCT)
• SCT will be the first gate before more extensive
  TE
• necessary, but not sufficient, for
  interoperability and mission effectiveness
• performed on distributed net-centric test-bed
  infrastructure
• Global Information Grid (GIG)
• provide on-site or downloadable SCT packages
• emphasizes services rather than systems
• testing of Service Oriented Architecture (SOA)
• GIG enterprise services

4
JITC is the Responsible Test Organization for
Tactical Data Link Standards

• Joint Interoperability Test Command (JITC) is
  responsible for ensuring systems that implement
  Tactical Data Link (TDL)
• are interoperable and
• in compliance with the applicable joint standards
  
• This is accomplished by conducting the following
  types of tests
• Joint / NATO /Combined Interoperability
• Performance Assessment in Operational
  Environments
• Standards Validation
• Standards Conformance
• JITC employs a variety of tools that provide its
  analysts the ability to evaluate TDL system
  performance in both the lab and live
  environments.

source http//jitc.fhu.disa.mil
Link 11/11B/16 and Variable Message Format (VMF)
5
Link-16 Challenges to Implementation and Testing

• Joint Single Link Implementation Requirements
  Specification
• JSLIRS is an evolving standard (MIL-STD-6016c)
  for tactical data link information exchange and
  networked command/control of radar systems
• Presents significant challenges to automated
  conformance testing
• The specification document states requirements in
  natural language
• open to ambiguous interpretations
• The document is voluminous
• many interdependent chapters and appendixes
• labor intensive and prone to error
• potentially incomplete and inconsistent.
• Problem how to ensure that a certification test
  procedure
• is traceable back to specification
• completely covers the requirements
• can be consistently replicated across the
  numerous contexts
• military service, inter-national, and commercial
  companies

6
Benefits of Formalization and Automation

• Provides traceability to original specification
• Reduces ambiguity from textual specification
• Facilitates integrating Modeling Simulation
  into the testing process
• Enables testing of complex
• Standards
• Systems
• Functions
• Families of systems

7
SIAP/IABM Successor to Link-16

• SIAP (Single Integrated Air Picture) Goal
  Improve the adequacy and fidelity of information
  to form a shared understanding of tactical
  situation
• Integrated Architecture Behavior Model (IABM)
  requires that all sensors utilize a standard
  reference frame for conveying information about
  the location of targets.
• Developed by the Joint SIAP System Engineering
  Organization (JSSEO), Arlington, Va., a
  sub-office of the Assistant Secretary of the Army
  for Acquisition, Logistics and Technology.

source http//www.navyleague.org/sea_power/mar_04
_18.php
8
Automated Test Case Generator (ATC-Gen)

• IABM is an extension of Link-16 developed in HLA
  environment and requires HLA simulation-based
  testing
• JITC has taken the initiative to integrate
  modeling and simulation into the automation of
  the testing process
• Funded the development of Automated Test Case
  Generator (ATC-Gen) led by ACIMS using DEVS
  (Discrete Event System Specification) technology.
  
• In RD of two years, proved the feasibility and
  the general direction
• First major test of IABM Config05 ATC-Gen is
  the basis for testing link-16 and extended IABM
  requirements

9
Discrete Event Nature of Link-16 Specification
10
ATC Gen Goals Approach

• Goals
• To increase the productivity and effectiveness
  of standards conformance testing (SCT) at Joint
  Interoperability Test Command (JITC)
• To apply systems theory, modeling and
  simulation concepts, and current software
  technology to (semi-)automate portions of
  conformance testing

Objective Automate Testing
Capture Specification as If-Then Rules in XML
Analyze Rules to Extract I/O Behavior
Synthesize DEVS Test Models
Test Driver Executes Models to Induce Testable
Behavior in System Under Test (SUT)
Interact With SUT Over Middleware
11
ATC Gen Tool

• Standard to XML Translation
• Analyst interprets the requirements text to
  extract state variables and rules, where rules
  are written in the form
• If P is true now Condition
• Then do action A later Consequence
• Unless Q occurs in the interim Exception
• Dependency Analysis Test Generation
• Dependency Analyzer (DA) determines the
  relationship between rules by identifying shared
  state variables
• Test Model Generator converts Analyst defined
  test sequences to executable simulation models
• Test Driver
• Test Driver interacts with and connects to SUT
  via HLA or Simple J interfaces to perform
  conformance testing
• Validated against legacy test tools

12
Capturing the Link-16 Standard in XML
Repository
13
Rule AnalysisExtract I/O Behaviors
RULE ANALYSIS
RESULTS
14
Test SequenceManually Derive Create Paths
Automatic Generation of Path File (Text)
ResponderNnot specifiedgt D.1.3.1.9ND.1.3.1gt D.1.3.
3.10.1a, D.1.3.3.13.1ND.1.3.1gt D.1.3.12.1ND.1.3.1gt

15
Test SequenceValidation Generation
Completed Path File
Repository
Appendix P
Appendix U
Section 4.11.13
Appendix D XML FILES
GENERATED TEST SEQUENCE (XML)
DEPENDENCY ANALYZER Validates Test Sequence
16
Test CaseGeneration
Test Sequence with Assigned Values
TEST CASE GENERATOR
GENERATED TEST CASE (XML)
17
Test ModelValidation Generation
Test Case
TEST MODEL GENERATOR
GENERATED TEST CASE MIRROR (XML)
Test Model
include "hierSequence.h" include
"PPLI.h" include "RemoteTNdrop.h"   const port_t
hierSeqDigraphstart0 const port_t
hierSeqDigraphinJmsg1 const port_t
hierSeqDigraphpass2 const port_t
hierSeqDigraphoutJmsg3   hierSeqDigraphhierS
eqDigraph()staticDigraph() PPLI pp new
PPLI() add(pp) couple(this, this-gtstart,
pp, pp-gtstart) couple(pp, pp-gtoutJmsg, this,
this-gtoutJmsg) RemoteTNdrop p1 new
RemoteTNdrop() add(p1) couple(this,
this-gtstart, p1, p1-gtstart) couple(this,
this-gtinJmsg, p1, p1-gtinJmsg) couple(p1,
p1-gtoutJmsg, this, this-gtoutJmsg)
18
Test ModelExecution
Test Model (C)
include "hierSequence.h" include
"PPLI.h" include "RemoteTNdrop.h"   const port_t
hierSeqDigraphstart0 const port_t
hierSeqDigraphinJmsg1 const port_t
hierSeqDigraphpass2 const port_t
hierSeqDigraphoutJmsg3   hierSeqDigraphhierS
eqDigraph()staticDigraph() PPLI pp new
PPLI() add(pp) couple(this, this-gtstart,
pp, pp-gtstart) couple(pp, pp-gtoutJmsg, this,
this-gtoutJmsg) RemoteTNdrop p1 new
RemoteTNdrop() add(p1) couple(this,
this-gtstart, p1, p1-gtstart) couple(this,
this-gtinJmsg, p1, p1-gtinJmsg) couple(p1,
p1-gtoutJmsg, this, this-gtoutJmsg)
TEST DRIVER
SYSTEM UNDER TEST
19
Extending MS-Based TE to the GIG

• Migrate ATC-Gen capability to the GIG-SOA
• accessible to all developers requiring Link-16
  conformance testing and certification
• enable self-tests for Link-16 standards
  conformance
• expose Link-16 XML-rule repository as authorized
  reference model
• register ATC-Gen test capabilities as web
  services
• Extend MS-Based automated testing
• provide mission thread testing for GIG deployable
  services
• certify transitions through the stages of DISAs
  Federated Distributed Certification Environment
• Expose MS-based TE assets as services
• experimental frame components generators,
  acceptors, transducers
• exploit DEVS reusability support
• characterize services for discovery and reuse

20
Link-16 XML Repository Web-Service Asset

• Organized according to MIL-STD-6016C
  macro-structure hierarchy
• Folders store aggregations/abstractions of lower
  level rules
• MIL-STD-6016C Reference Model
• Removes ambiguity
• Annotates problems areas, improving the ability
  to find and fix issues
• Provides organization for indexing states, rules,
  and variables
• Supports test generation and executable rule
  construction

21
Transferring DEVS-based Testing Methodology to SOA

• DEVS-based Testing Methodology readily migrates
  to SOA environment as shown below
• In particular, the ATC-Gen Test Driver can be
  offered as a service for Link-16 conformance
  self-test
• Further, the methodology can be applied to other
  test and evaluation contexts

22
Automating Mission Thread/Vignettes for GES
Testing

• Objective automate to the maximum extent the
  definition of mission threads/vignettes and the
  generation of test services based on them
• Enable COI and community wide service users and
  developers to easily write threads/scenarios in
  (restricted) Natural Language form
• Interacting with such users/developers,
  implement these scenarios with DEVS test
  federation models that can execute over Core ES
• Mission thread/Vignette scenario testing applies
  objectives-driven testing to all service
  levels and throughout service offering life
  cycle
• Repository can support re-use and re-application
  for regression testing to preserve service
  behavior under continual change in SOA environment

23
Mission Thread Testing Applies to all Service
Levels and Throughout Service Offering Life Cycle
Mission Threads enable objectives driven
testing throughout service offering life-cycle
24
NLP Mission Thread Scenario Generation
Restricted Natural Language form of Mission
Thread Scenario Specification
JTAC working with FourthInfDiv . USNAircraft
gets deconfliction from USMCAircraft
Transform to System Entity Structure
Scenario in XML
Natural Language Parser
(actor JTAC, action work, actee
FourthInfDiv) . (actor JTAC, action
requests, message ImmediatCAS, actee
DirectAirSuppAirborne) .
Transform to DEVS
Parser outputs tagged equivalent of input
DEVS Test Driver for NCES Mission Threads
25
Example Joint Close Air Support Scenario
26
NLP Test Generation Example Joint Close Air
Support Scenario
JTAC works with ODA! JTAC is supported by a
Predator! JTAC requests ImmediateCAS to AWACS
! AWACS passes requestImmediateCAS to CAOC!
CAOC assigns USMCAircraft to JTAC! CAOC sends
readyOrder to USMCAircraft ! USMCAircraft sends
sitBriefRequest to AWACS ! AWACS sends sitBrief
to USMCAircraft ! USMCAircraft sends
requestForTAC to JTAC ! JTAC sends TACCommand to
USMCAircraft ! USMCAircraft sends
deconflictRequest to UAV! USMCAircraft gets
targetLocation from UAV!!
NCES
27
DEVS-based Web-Services Testing
28
Demohttp//acims9.acims.arizona.edu
Web Service Find aircraft available for CAS
given coordin-ates and time frame.
29
MS-Based TE Services on the SOAExperimental
Frame Repository

• Compose EF from generator, acceptor, and
  transducer components in repository
• Characterize functionalities using MS Theory and
  WSDL
• Support composability based on migration of DEVS
  technology to SOA

Request
Generator
Acceptor Repository
Transducer Repository
Repository
30
Example Synthetic Environment Repository
Experimental Frame Partial Order
Experimental Frames needed for Testing Objectives
Experimental Frame- Model Applicability
Relations
Environmental and Standards Reference Models
available for use in Testing
31
Summary and Conclusions

• There is an acute need for a new Net-centric
  testing paradigm at the enterprise level where
  joint and coalition operations are conducted
  (Editorial, ITEA Jnl, Sept, 2005)
• Net-Centric TE places an increased emphasis on
  standards conformance self-testing over the
  GIG-SOA and emphasizes services rather than
  systems
• Due to its formal and abstract basis, the DEVS
  functionality migrates quite directly to execute
  over the web-services middleware based on XML and
  SOAP
• This extension enables semi-automated
  construction of DEVS test federations for
  verification testing of a wide variety of
  proposed web services
• ATC-Gen Link-16 and other TDL standards
  conformance self testing
• Mission thread or vignette scenario specification
  using natural language input can replace Link-16
  style standards formalization as the source input
  for test case generation
• MS-based TE services can exploit experimental
  frame methodology

32
Bernard P. Zeiglerzeigler_at_ece.arizona.edu
ACIMSwww.acims.arizona.eduJITCjitc.fhu.disa.mil

Contact
More information
33
Joint Interoperability, Assured Security, Best
Value, Customer Success
Global Net-Centric Solutions -- The Warfighter's
Edge