Title: The MPI Testing Tool MTT
1The MPI Testing Tool (MTT)?
- Ethan Mallove - Sun Microsystems
- Jeff Squyres - Cisco Systems
- Josh Hursey - Indiana University
- Andrew Lumsdaine - Indiana University
- EuroPVM/MPI 2007
2Overview
- Why MTT?
- MTT Usage
- Configuration
- Test specification
- Test execution
- Reporting Tool
- Case study
3The problem with testing MPI
- Portable MPI implementation needs regular
(correctness and performance) testing - On multiple platforms / architectures / operating
systems - Across multiple organizations / environments
- With many different test suites
- Exercising hundreds of run-time configuration
options - Require repeatability
4Available testing frameworks
- TET no software retrieval
- Perfbase storage model did not meet our
requirements - DejaGNU test creator
- Others Dart, Boost, Mozilla Tinderbox, Testopia,
Buildbot, etc.
5Testing Flow
- MPI Get obtain the MPI implementation(s) to test
- MPI Install install the MPI implementation(s)?
- Test Get obtain the test suite(s) to run
- Test Build build the test suite(s) against each
MPI implementation - Test Run run all the test suite(s) against each
MPI implementation
6Testing Flow (cont.)?
7MTT Configuration
- INI-style text file
- MPI Details section - how to run a job under this
MPI - MPI get/install sections - how to obtain /
install MPI - Test get/build/run sections - how to obtain /
install / what tests to run - Reporter section - what to do with the results
8MTT Funclets
- Perl-like functions used in the INI file to do
the following - Multiplicative effectexec mpirun -np _at_np_at_
executable()np enumerate(2, 4, 8, 16)? - Logicpass eq(wifexited(), 0)timeout
max(30, multiply(10, test_np()))?
9MTT Design
- The client is written in Perl
- Modular plug-in framework
- Plug-ins for
- MPI specific functionality
- Test suite specific functionality
- Reporter specific functionality
10MTT Test Specification
- Plug-ins carry out
- Complex builds
- Test grouping
- Control over pass/fail/time-out conditions
11MTT Test Execution
- Managed via MPI Details INI sections
- Flexible enough to test any MPI
- Examples
MPI Details MPICH exec mpirun -np 2
executable()? MPI Details Open MPI exec
mpirun --mca btl _at_btls_at_ -np np()
executable()? btls enumerate("self,sm,tcp",
"self,sm,openib")?
12MTT Reports
Web-driven reporting tool covers many use cases
for searching on test results
- High-level summaries
- Root causing
- Performance results
- E-mailed reports and permalinks
13MTT Reporting Tool
- Web-driven tool
- "Drill down" and color-coded approach to managing
sea of test results - Summary and Detail views
- Results are never discarded
- Flexible search engine
14Case Study The Open MPI Project
- Multitude of runtime parameters
- Many configurations
- Many platforms
- Many test suites
- etc.
The combinatoric explodes ...
15Case Study The Open MPI Project (cont.)?
- Leverage a pool of software/hardware testing
resources - Unite the community of developers and testers
- Central store of test results makes collaboration
easier
16MTT Future Work
- Heterogeneous testing environments
- "Disconnected" scenarios (e.g., build on one
machine, run in another)? - Build parallelism into the testing framework
- Generalize for other software
17Try MTT Yourselves!
- http//www.open-mpi.org/projects/mtt
18MTT Reports
19Summary
- Summarize the most important points of your
presentation here
20Discussion
- Present some ideas concerning the main points of
your presentation - Provoke questions and comments from your audience