SSS Validation and Testing

1
SSS Validation and Testing

• September 11, 2003
• Rockville, MD
• William McLendon
• Neil Pundit
• Erik DeBenedictis

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energy under contract DE-AC04-94AL85000.
2
Overview

• APItest
• Release Testing Experiences at Sandia
• Status daemon

3
Distributed Runtime System Testing

• Complex system of interactions
• Approach to testing
• Component Testing
• Benchmarks
• Performance / Functionality
• Operational Profile
• Stress Testing
• Users expect a high-degree of quality in todays
  high end systems!

4
APItest
5
APITEST - Overview

• Unit-testing tool for network components
• Targeted for networked applications
• Extensible framework
• Dependency calculus for inter-test relationships
• Scriptable Tests (XML Schema Grammar)
• Multi-Protocol Support
• TCP/IP, SSSLib, Portals, HTTP

6
Accomplishments Since Last Meeting

• Spent a week at Argonne (July)
• Major rework of framework of APItest
• Individual tests are atomic.
• Framework handles the hard work of checking
  tests, dependencies, and aggregate results.
• Extensibility
• New test types are easy to create
• Dependency System
• Define relationships as a DAG encoded in XML.
• Boolean dependencies on edges.

7
Supported Test Types

• sssTest
• use ssslib to communicate with ssslib enabled
  components
• shellTest
• execute a command
• httpTest
• ie. app testing web interfaces (ala Globus, etc)
• tcpipTest
• raw socket via tcpip transmission.

8
APITEST supports Regular Expressions

• Regular expression matching in tests

Sent to service-directoryltget-locationgtltlocation
component/gtlt/get-locationgt Expected
Bufferltlocationsgt ltlocations
componentservice-directory/gt ltlocations
componentevent-manager/gtlt/locationsgt Regular
Expressionltrecvbuf typeREGEXPgt
ltlocationsgt(\sltlocation\scomponent\S\s/gt\s
)lt/locationsgtlt/recvbufgt
9
Creating New Test Types is Easy
A simple test that will always pass class
passTest(Test) __attrfields__name typemap
dependenciesTODependencies def
setup(self) pass def execute(self,
scratch) self.expectfoo (REGEXP,
a) self.responsefoo a
10
Matching and Aggregation

• An individual test can be executed many times in
  a sequence.
• PASS/FAIL can be determined based on the percent
  of runs that matched.
• Percent Match can be specified as a range as
  well.
• Expected result is specified as a regular
  expression (REGEX) or a string for exact matching
  (TXTSTR)
• Notation
• Mminmax - Percent matching.
• min/max bounds on of tests where actual and
  expected results match.
• If the actual number of tests is within the range
  specified the test will PASS, otherwise it will
  FAIL.

11
Test Dependencies
T iff A4090OR B00
T iff (A100 ? B90) ?C
A
B
M4090
M00
?
?
M4090 gt 40 and lt 90 of test runs matched
12
An Example Dependency
ltdependenciesgt ltANDgt ltdependency
nameA' minPctMatch100'/gt ltORgt
ltANDgt ltdependency nameB'
minPctMatch100'/gt ltdependency
nameC'/gt lt/ANDgt
ltdependency nameD maxPctMatch0/gt
lt/ORgt lt/ANDgt lt/dependenciesgt

• (A100 ? ((B100100 ? C) ? D0 ))

13
An Example Test Sequence
test daemon
M30
reset daemon
M
M
test other stuff
14
Standard Test Metadata Attributes
15
Example Scripts

• A simple shell execution test
• ltshellTest nametest1 numReps1
  preDelay5 postDelay5.3
  commandls -ltr/gt
• Test with a dependency and stdout matching
• ltshellTest nametest2 commandapitest.py
  --testgt ltoutput formatREGEXP
  typestdoutgt.stdout.lt/outputgt
  ltdependenciesgt ltdependency nametest1
  minPctMatch100.0/gt lt/dependenciesgtlt/shellTe
  stgt

16
APItest Output
iterations test name matched Pass/Fail message
---------- --------- --------- --------- ---------
- 1 of 1 A 100.00 PASS 1 of
1 K 100.00 FAIL m0.0 0.0 1
of 1 J 0.00 FAIL m90.0
90.0 5 of 5 M 100.00 PASS 1
of 1 L 100.00 FAIL m0.0
0.0 1 of 1 N 100.00 PASS 0
of 1 T DEPENDENCY FAILURE(S) F
expected 0.0 90.0, got 100.0 J expected
90.0 100.0, got 0.0 0 of 1 R
DEPENDENCY FAILURE(S) J expected 90.0
100.0, got 0.0 K expected 0.0 90.0,
got 100.0 1 of 1 S 100.00 PASS
1 of 1 U1 100.00 PASS 1 of
1 U2 100.00 PASS 0 of 1 U3
DEPENDENCY FAILURE(S) N expected 0.0
90.0, got 100.0 S expected 0.0 90.0,
got 100.0 1 of 1 U4 100.00 PASS
17
sssTest outputs from Chiba City
iterations test name matched Pass/Fail
message ---------- --------- ---------
--------- ---------- 1 of 1
add-location 100.00 PASS 1 of 1
QuerySDComps 100.00 PASS 1 of 1
QuerySDHost 100.00 PASS 1 of 1
QuerySDProtocol 100.00 PASS 1 of 1
QuerySDPort 100.00 PASS 1 of 1
del-location 100.00 PASS 1 of 1
val-removal 100.00 PASS
iterations test name matched Pass/Fail
message ---------- --------- ---------
--------- ---------- 1 of 1 sss-getproto
100.00 PASS 1 of 1 sss-getport
100.00 PASS 1 of 1 sss-gethost
100.00 PASS 1 of 1 sss-getcomp
100.00 PASS 1 of 1 sss-getproto
100.00 PASS 1 of 1 sss-getport
100.00 PASS 1 of 1 sss-gethost
100.00 PASS 1 of 1 sss-getcomp
100.00 PASS
18
Release Testing
19
Tales from Cplant Release Testing

• Methodical execution of production jobs and 3rd
  Party benchmarks to identify system
  instabilities, enabling them to be resolved. Ie
• Rapid job turnover rate (caused mismatches
  between scheduler and allocator)
• Heavy io (I/O which passes through launch node
  process instead of directly to ENFS yod-io)
• Wrapping above codes into Ctest framework to
  enable portable compile, launch, and analysis of
  synthetic workloads

20
Ctest

• Extension of Mike Carifios work
• Presented at the SciDAC meeting in Houston during
  Fall of 2002
• Make structure that holds a suite of independent
  applications.
• Tools to launch as a reproducible workload.
• Goal 30 users and 60 concurrent apps

21
Sample Load Profile on CPlant
22
Issue Tracking

• SNL uses a program called RT
• Centralized repository for issue tracking helps
  give an overall picture of what problems are.
• Helps give summary of progress.
• Bugzilla is on the SciDAC SSS website
• http//bugzilla.mcs.anl.gov/scidac-sss/
• Whos using it?

23
Status Daemon
24
Status Daemon

• Highly configurable monitoring infrastructure for
  clusters.
• Does not need to run daemon on the node you are
  monitoring.
• XML configurable
• Web interface
• Cluster Aware
• Used on CPlant production clusters
• James Laros (jhlaros_at_sandia.gov)

25
Status Daemon Communication
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
Summary

• New Hire
• Ron Oldfield
• APItest functionality and flexibility increases
• Release testing experience
• Status Daemon

33
Plans

• APItest
• User / Programmer Manuals
• User Interface
• GUI? HTTP?
• daemon mode for parallel testing mode
• DB Connectivity
• Test Development
• ssslib event tests
• HTTPtest work
• ptlTest (SNL)
• SWP Integration
• Port SWP to Chiba for SC2003?

34
Questions?
35
END OF PRESENTATION
36
APITEST Component Architecture
37
Encoding XML into XML CDATA
XML parsers will interpret the lt character as a
newElement, NOT as a character buffer. So,we
must trickour SAX parse by using lt instead of
lt
ltssslibTestgt ltnamegttest1lt/namegt
ltdestinationgtsdlt/destinationgt
ltsendbufgt ltget-locationgt ltlocation
component''/gt lt/get-locationgt
lt/sendbufgt lt/ssslibTestgt