OVAL

1
OVAL

• A testing tool for CMS software
• http//polywww.in2p3.fr/cms/software/oval

2
General Description

• Original goal help to detect automatically the
  changes in the physics performances of the CMS
  software used for data production.
• Current features of the tool compile the
  provided test programs, run them in specified
  runtime conditions (shell variables auxiliary
  files), compare the output with references.
• User interface is CVS-like, configuration files
  are XML-like. It is written in PERL and can be
  configured for any build system (scram, make).

3
Outline

• General description.
• Configuration OvalFile.
• Commands build/run/diff.
• Results log files.
• Other features.
• Summary.
• Plans.

4
Configuration OvalFile

• ltdiffline expr"OVAL"gtltdiffnumber expr"OVAL
  NUM . is (.)" tolerance"5"gt
• ltfile name".orcarc"gtMaxEvents
  500RandomSeeds 0 3lt/filegt
• ltvar name"FEDERATION" value"cmsuf01"gt
• ltenvironment name"pt15"gt ltvar
  name"DATASET" value"eg_ele_pt15"gt ltprogram
  name"Clusters.cpp"gt ltprogram
  name"Electrons.cpp" args"-geo
  detailed"gtlt/environmentgt
• ltenvironment name"flow"gt ltvar
  name"DATASET" value"jm_minbias"gt ltprogram
  name"EnergyFlow.cpp"gtlt/environmentgt

5
Commands build/run/diff

• cms038ElectronPhotongt ls
• Makefile
• CVS
• Clusters.cpp
• Clusters.ref
• Electrons.cpp
• Electrons.ref
• EnergyFlow.cpp
• OvalFile

6
Commands build/run/diff

• cms038ElectronPhotongt oval prod
• Clusters build, run, diff.
• Electrons build, run, diff (DIFFS).

7
Commands build/run/diff

• cms038ElectronPhotongt ls
• Makefile
• CVS
• Clusters.cpp
• Clusters.log
• Clusters.ref
• Electrons.cpp
• Electrons.log
• Electrons.ref
• EnergyFlow.cpp
• OvalFile
• electrons.hbook

8
Results log files

• oval build
  
• oval build make Electrons
• oval build
  
• /usr/local/gcc-alt-2.95.2/bin/c -O2 -pthread
  -rdynamic
• -L/afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__
  2.2
• -L/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.
  2
• -L/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.
  95.2/3.6.2/lib
• -L/afs/cern.ch/cms/cmsim/cms122/lib/i386_linux22
  
• -L/cern/2001/lib
• Electrons.o -o Electrons
• -lElectronFacilities -lEgammaH4Support
  -lEgammaL1Tools
• -lmathlib -lblas -lkernlib -lpythia6152 -lz
  -lnsl -ldl -lg2c -lm

9
Results log files

• oval run
  
• oval run LD_LIBRARY_PATH
• oval run /afs/cern.ch/user/c/chamont/ORCA_5/li
  b/Linux__2.2
• oval run /afs/cern.ch/cms/Releases/ORCA_5/lib/
  Linux__2.2
• oval run /afs/cern.ch/sw/lhcxx/i386_linux22/Ob
  jectivity/5.2.1/lib
• oval run FEDERATION cmsuf01/cms/jet0900/jet0
  900.boot
• oval run DATASET eg_ele_pt15
• ...
• oval run .orcarc
• oval run MaxEvents 500
• oval run RandomSeeds 0 3
• oval run
  
• Welcome to COBRA
• ...

10
Results log files

• oval diff
  
• oval diff diff line /OVAL/
• oval diff diff number /OVAL NUM . is
  (.)/ 5
• oval diff
  
• ref1452 ! log2053
• lt OVAL 12 electrons
• ---
• gt OVAL 11 electrons
• ref1972 ! log2592 (gt5)
• lt OVAL NUM energy mean is 29.7275
• ---
• gt OVAL NUM energy mean is 27.4728

11
Other Features

• Oval is not restrained to programs. For complex
  testing tasks, especially when databases are
  involved, one can also handle some scripts (no
  build step needed).
• The same program or script can be executed
  several times with different arguments, and/or
  within different environments.
• If there are many tests, one can organize them in
  a hierarchy of Unix directories, and create
  OvalFiles at any level.
• The results of an oval command can be mailed to
  watchers as defined in the configuration.

12
Summary

• Scan the output, whatever it is.
• Can be combine with any build system.
• Very easy to use, except the tuning of regular
  expressions (more examples available soon).
• No helper code provided. The users still have to
  write their test programs.
• CMS use it for regression tests some developers
  run it daily on their code (kind of extreme
  programming !)
• LCG plans to use it together with CppUnit.

13
Plans

• Improve infrastructure support
• improve internal documentation,
• add a tutorial,
• progressively move to LCG savannah server.
• React to users feedback
• enable the users to provide their own diff
  command,
• add a reuse mechanism for the environments,
• automatic generation of trivial test programs for
  a given set of header files.