SciDAC SSS Quarterly Report

1
SciDAC SSS Quarterly Report

• Sandia Labs
• May 10, 2005
• William McLendon
• Ron Oldfield
• Neil Pundit

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energys National Nuclear Security
Administration under contract DE-AC04-94AL85000.
2
APItest - Overview

• Current Release v1.0
• Test Driver tool for testing software
• Ordered tests
• API Testing
• Available for download
• ftp//ftp.sandia.gov/outgoing/apitest/

2
3
Work Since Previous Meeting

• New Look via Cascading Style Sheets
• Can be turned off if needed.
• New Capabilities
• PASS/FAIL status with batch files.
• Some new XML options for batch tests.
• Better reporting of parse errors in XML
• (very) preliminary connectivity to MySQL DB
• Saves results only, no browse capability.
• Finished User Guide
• Preliminary RA approval
• Slight modifications needed and its out.
• Bug Fixes

3
4
APItest - Features

• Open Source (LGPL)
• Scripted Test Driver
• Runs XML test scripts.
• Written in Python
• Portable
• Object Oriented
• Customizable
• Built in Test Types
• Shell
• Script
• SciDAC SSS (ssslib)
• TCP/IP
• XML Error Helper
• Help find XML Errors

• Conditional execution of tests test ordering.
• Some Test Features
• SUID Capability
• Test Timeouts
• Pattern matched results
• Batch Status Conditions
• User interfaces
• Interactive via browser.
• Command line mode.
• Limited MySQL DB Support
• (under development)

4
5
APItest Package Requirements

• Python 2.3 or greater
• lower versions can work but the setup script
  requires distutils, which is a python 2.3
  package.
• ElementTree (www.effbot.org)
• Twisted (www.twistedmatrix.com)
• New version of 2.0.0 has changed the twisted
  library and added new dependencies. Discussed on
  next slide.
• SSSlib (ftp//ftp.mcs.anl.gov/pub/sss/)
• optional, for SSSlib tests.
• MySQL
• optional, for MySQL database connectivity.

5
6
Recent Changes in the Twisted Library

• With Twisted-2.0.0 the distribution has been
  split up.
• Twisted-2.0.0 Prerequisites
• Python 2.3 or greater
• ZopeInterfaces-3.0.1 or greater
• TwistedWeb Prerequisites
• Twisted-2.0.0 must be installed first.
• APItest still works with Twisted-1.2.0
• Still available at www.twistedmatrix.org with a
  little digging.

6
7
Batch File PASS/FAIL

• 2 New additions to Batch files
• New ATTRIBUTE to a element
• mustPass TRUE / FALSE
• Determines if a test counts towards batch file
  pass/fail stuats.
• If set to true (default), this test MUST PASS or
  the batch from which it is called will FAIL.
• If set to Fail, this test will not be considered
  in batch status determination.
• Must provide a element to set this
  attribute.
• New Element
• Changes default for mustPass.

7
8
MySQL Connectivity

• Started adding MySQL connectivity
• Very new and certainly in testing.
• Some new packages required to make it work
• MySQL
• MySQLdb python modules
• Saves results to DB only, no browse capability
  yet.
• Command line driven
• Will show up in help and the userguide.
• More work than it seems like. Need to rewrite a
  couple of things in APItests jobManager to make
  things work the right way.

8
9
New Look and Feel

• Added a new look to the gui browser.
• Tabbed menu system
• Cascading Style Sheets (CSS)
• There are some compatibility issues, not all
  browsers render it necessarily the same.
• Added a compatibility mode option incase CSS
  messes up.
• Add nocss to command line.

9
10
APItest Main Screen

• Lists available tests
• Color codes tests
• Batch files
• Individual tests
• Invalid test files.
• Organized by directory
• Details of tests can be viewed by clicking the
  View Test button.
• Click SUBMIT to execute selected tests.

10
11
Running A Test

• Session screen shows the currently running test.
• Tests show their PASS/FAIL status.
• Batches show an overall PASS or FAIL and some
  statistics.
• Total of tests in batch
• Total that passed.
• Total that failed.

11
12
Viewing Test Results

• Browsing a tests result screen shows what
  happened.
• Execution summary
• Expected and Actual outputs
• Dependency info (when applicable)
• These results are also saved to disk in XML

13
Dependency Failure

• Viewing test results when a dependency failure
  shows what failed.
• Each parent test is listed with Expected and
  Actual status.
• Failed dependencies are highlighted to make them
  stand out.

13
14
Helping Fix Bad Tests

• APItest provides debugging assistance for test
  scripts.
• Click View Test from the Main screen for the
  test listing
• Lines are numbered.
• The error is shown via ERROR
• In this case, is invalid XML. Replace with
  amp

14
15
Terminal Execution

• APItest can be run at the command line.
• Useful for running a test suite unattended such
  as through a scheduler.
• Results are saved and can be browsed later via
  the browser.
• A view-only mode is available that only can
  browse saved results.

15
16
APItest - Ongoing TODO

• Bug Fixing
• User Guide
• Preliminary RA given.
• DB Connectivity
• Make MySQL connectivity more complete.
• Validation via file matching
• Configuration file
• More SSS tests
• Test Developer GUI
• Remote test execution
• Additional native test types
• HTTP, XMLRPC
• Session management and security (Cookies?)

16
17
(No Transcript)
18
APItest - Execution

• There are two ways to execute APItest
• Command Line Only
• apitest -f test_file.aptapb
• Web browser GUI (Interactive)
• apitest httpd
• -i --iroot root directory to look for tests
• -p --port port number (default 2112)
• -h --host host (default localhost)
• Running with --help will display command line
  usage.

18
19
SSS - Service Directory Startup Test
20
Dependency Tree for APItest

• Python version 2.3
• ZopeInterfaces-3.0.1
• Twisted-2.0.0
• TwistedWeb-0.5.0
• ElementTree
• ssslib (optional)
• MySQL (optional)

21
APItest - Test Types

• CMD
• Execute a command via the command line.
• SCRIPT
• Execute a script.
• SSS
• Execute a ssslib communication.

22
APItest - Example CMD Test
Example test. Runs ls
-lr from /tmp, expects 0 exit status and any
input. wdir"/tmp" sh"ls format"literal"-l name"argument" format"literal"-r

.t format"literal" / name"status" format"literal"0

23
APItest - Example Script Test
Timeout test
timeout"4" ash" wdir"/tmp" for x in 1 2 3 4 5 6 7 8 9 10
do echo "timeout_test x date
sleep 1 done name"stdout" format"regexp".timeout.
/ format"literal"0

24
APItest - Example ENVVAR Test
This script executes but
adds some stuff to the environment. NOTE
setting envvar OVERRIDES any inherited
environment variables. type"script" interpreter"/bin/csh"!/bin/csh env
key"test_envvar_1"foobar name"envvar" format"literal" key"test_envvar_2"
foo format"regexp" .test_envvar_1fooba
r\n.test_envvar_2foo\n.

0

25
APItest - Example SUID Test

gid"500" !/usr/bin/env python import os
print os.getuid() print os.getgid()
/tmp/te
stuser/ -l/input format"regexp"27030\n500

0

26
APItest - Example SSS Test
Sample sss query of the
service directory destination"service-directory" name"sendbuf" ltget-locationltlocation
component'service-directory' host'' port''
protocol'' schema_version''
tier''/lt/get-location
.

27
Old Graphics
28
APItest - GUI (Top Page)

• Top level index page for APItest
• APItest recurses from the testroot to find
  tests.
• File Extensions
• TEST .apt
• BATCH .apb
• Batch tests shown in bold
• Organized by directory

29
APItest - GUI (Test Run Execution)

• Quick status view of test execution.
• This screen loads when the EXECUTE button is
  selected.
• Color coded for easy viewing.
• Currently running test is indicated.
• Refreshes every 15 seconds until tests complete.

30
APItest - GUI (Test Run Summary)

• Summary of test results
• Color coding
• Batch Features
• Cumulative summaries
• Total of tests
• that passed (green)
• that failed (red)
• Hyperlinks to see more details about a test or
  batch.

31
APItest - GUI (Test Details - PASSED)

• Example of a test that was executed and passed.
• Summary Data
• Dependency Data
• Only shown if test was run as part of a batch.
• Output
• Side-by-side comparison
• Dependent on type of test.

32
APItest - GUI (Batch Summary)

• Clicking a hyperlink into a batch test gives us a
  summary view.
• In this example, one test failed a dependency
• FAILDEP in the status column indicates this.
• Clicking on the scriptTest2.apt link will show us
  more data about this test.

33
APItest - GUI (Test Details - FAILDEP)

• Detailed view of scriptTest2.apt from previous
  slide.
• Failed a dependency on cmdTest2.apt
• Expected FAIL
• Actual PASS

34
APItest - GUI (List of Test Runs)

• Clicking the View All Runs button gives this
  screen.
• Test Runs are sorted by date and time.
• Click links to view details of a particular run.