Tutorial on conventions, policies, CMT patterns in Atlas

1
Tutorial on conventions,policies, CMT
patternsin Atlas

• 2003 December 3rd
• Atlas software workshop
• Christian Arnault

2
Tutorial plan

• The (offline) Atlas policies
• The installation area structure
• The distribution kit
• Some basic techniques for implementing the
  policies
• The standard macros
• The CMT patterns
• Novelties in CMT v1r14 recent patches
• Making a package properly structured
• Building libraries, applications, test programs
• Running documentation generator and quality tools
• Installing files into the installation area
• Setting up for the distribution kit
• Discussion

3
The offline software base

• The software is divided into
• Internal software
• Gaudi
• External software
• Atlas private copies of external products
• LCG software
• HEP software
• Public domain software

Installation area
Release area
sources
CVS
cmt
Work area
binaries
generated
External
External
External
External
cmt
External
4
The distribution structure (1/2)

• Makes the software base transportable
• Relocateable relatively to SITEROOT
• Groups all external software in one single
  location (SITEROOT)
• Keeps installation area(s) for all
• binaries
• include files
• data and runtime files (joboptions, xml,
  scripts,)
• Does not need sources
• May be added later on for making source kits
• It is both a runtime and development kit
• Athena applications can be run straight from it
• No need to use TestRelease (although possible)
• Permits user development (build) of checked out
  packages
• Will soon become the base release structure even
  at CERN
• Permits (fast) intermediate builds on local file
  systems

5
The distribution structure (2/2)
SITEROOT
includes
7.0.2/InstallArea/... 0.12.0.1/InstallArea/... ...
Installation areas
binaries
generated
External
External
External
cern/2002/... root/... sw/lcg/... ...
External
External
7.0.2/Control/AthenaCommon/AthenaCommon-/cmt/...
7.0.2/External/AtlasROOT/AtlasROOt-/cmt/... 0.12.
0.1/GaudiSvc/v/cmt/... ...
6
reminder CMT Basics

• Getting CMT itself
• Current suggested version is CMT v1r14p20031120
• gt source /afs/cern.ch/sw/contrib/CMT/v1r14p2003112
  0/mgr/setup.csh
• http//www.cmtsite.org
• User guide
• http//atlas-sw.cern.ch/cgi-bin/cvsweb.cgi/checko
  ut/offline/
• AtlasDoc/doc/SwDevUserGuide/userguide.html

7
reminder CMT Basics macros (1/6)

• The standard CMT macros
• use_xxx (constructed by CMT)
• Groups all ltpackagegt_xxx macros for all used
  packages in the right order.
• Individual package contribution hidden by the
  no_auto_imports option of the use statement
• package
• version
• ltpackagegt_cmtpath
• ltpackagegt_offset
• application_suffix
• library_prefix
• library_suffix
• constituents

use_cflags C compiler flags use_pp_cflags
Preprocessor flags for C use_cppflags C
compiler flags use_pp_cppflags Preprocessor
flags for C use_fflags Fortran compiler
flags use_pp_fflags Preprocessor flags for
Fortran use_libraries List of library
names use_linkopts Linker options use_stamps
Dependency stamps use_requirements The set of
used requirements use_includes include search
paths use_fincludes include search paths
8
reminder CMT Basics - macros (2/6)

• The standard CMT macros
• use_xxx
• package (constructed by CMT)
• version (constructed by CMT)
• contains the current package name and version
• ltpackagegt_cmtpath
• ltpackagegt_offset
• application_suffix
• library_prefix
• library_suffix
• constituents

9
reminder CMT Basics - macros (3/6)

• The standard CMT macros
• use_xxx
• package
• version
• ltpackagegt_cmtpath (constructed by CMT)
• ltpackagegt_offset (constructed by CMT)
• contains the CMTPATH entry used to find the
  specified package and the offset in this
  directory
• application_suffix
• library_prefix
• library_suffix
• constituents

gt cmt show macro_value package
AthenaCommon gt cmt show macro_value
version AthenaCommon-01-04-13 gt cmt show
macro_value AthenaCommon_cmtpath
/afs/cern.ch/atlas/software/dist/nightlies/rel/at
lrel_2 gt cmt show macro_value AthenaCommon_offset
Control
10
reminder CMT Basics - macros (4/6)

• The standard CMT macros
• use_xxx
• package
• version
• ltpackagegt_cmtpath
• ltpackagegt_offset
• application_suffix (originally defined by CMT)
• Specifies the suffix used when generating the
  executable applications (defaults to .exe)
• library_prefix
• library_suffix
• constituents

11
reminder CMT Basics - macros (5/6)

• The standard CMT macros
• use_xxx
• package
• version
• ltpackagegt_offset
• ltpackagegt_cmtpath
• application_suffix
• library_prefix (originally defined by CMT)
• library_suffix (originally defined by CMT)
• Specifies the prefix and suffix to be added to
  library names
• Defaults are "lib" and empty to form
• libltnamegt.so
• libltnamegt.a
• constituents

12
reminder CMT Basics - macros (6/6)

• The standard CMT macros
• use_xxx
• package
• version
• ltpackagegt_offset
• ltpackagegt_cmtpath
• application_suffix
• library_prefix
• library_suffix
• constituents (constructed by CMT)
• Give the list of constituents (applications,
  libraries, documents) that will be built by
  default (associated with the all target)
• Can be modified (using macro_append,
  macro_remove) to control platform specificities
  or context management.

13
CMT basics - the patterns (1/2)

• What is a CMT pattern
• Its a parameterised collection of CMT statements
  frequently used to apply a convention, a policy,
  a complex mechanism
• It can be automated with the gobal option
• automatically applied to all packages that see it
• Defining a pattern
• pattern name stmt stmt ...
• pattern global name stmt stmt ...
• Composing statements may use parameters
• Standard parameters
• ltpackagegt
• ltversiongt
• ltPACKAGEgt
• User defined parameters
• Use the ltxxxgt notation
• Default value is an empty string
• Applying a pattern
• Provide the user defined parameter values
• apply_pattern name paramvalue paramvalue

14
CMT basics - the cmtpath patterns (2/2)

• What is a cmtpath pattern
• Its a special CMT pattern that will be applied
  for each entry of the CMTPATH list
• It is always applied (no need to explicitly apply
  it)
• It is applied in a global scope (outside of any
  package)
• Defining a cmtpath pattern
• cmtpath_pattern stmt stmt ...
• Composing statements may use only one parameter
  ltpathgt matching an entry in the CMTPATH list.

path_prepend PATH \ "ltpathgt/(cmt_installarea_p
refix)/tag/bin" \ path_prepend CLASSPATH \
"ltpathgt/(cmt_installarea_prefix)/share/lib"
\ path_prepend CLASSPATH \ "ltpathgt/(cmt_instal
larea_prefix)/share/bin" \ macro_prepend
cmt_installarea_linkopts "" \ Unix "
-Lltpathgt/(cmt_installarea_prefix)/(tag)/lib "
\ WIN32 " /LIBPATHltpathgt/(cmt_installarea_pre
fix)/(tag)/lib "
15
Novelties in CMT v1r14 recent patches

• Exclusion and selection patterns in constituent
  sources
• library A kltsel-regexpgt -xltexcl-regexpgt .cxx
• -kltregexpgt to select a subset of .cxx
• -xltregexpgt to exclude a subset of .cxx
• New show arguments
• gt cmt show use_paths ltpackagegt
• Display all possible paths towards a given used
  package
• gt cmt show cycles
• Display all use cycles

gt cmt show use_paths GaudiInterface IOVDbTestCondi
tions.CLIDSvc.GaudiInterface ...CLIDSvc.AthenaKe
rnel.GaudiInterface
16
CMT basics

• Links
• The Atlas user guide
• http//atlas-sw.cern.ch/cgi-bin/cvsweb.cgi/checko
  ut/offline/
• AtlasDoc/doc/SwDevUserGuide/userguide.html
• The HowTo document
• http//atlas-sw.cern.ch/cgi-bin/cvsweb.cgi/checko
  ut/offline/
• AtlasDoc/doc/sit/UserDev_HowTo.html
• The CMT documentation
• http//www.cmtsite.org/CMTDoc.html

17
Basic Atlas patterns

• Available as soon as one uses AtlasPolicy
• Global patterns
• The location of header files default_include_path
• Doxygen generation doxygen
• Coding style checker checkrules
• cmtpath_patterns

path_prepend PATH ltpathgt/(cmt_installarea_prefix)
/share/bin \ set_prepend JOBOPTSEARCHPATH \
",ltpathgt/(cmt_installarea_prefix)/jobOptions" 
\ document install_includes install_includes
\ macro_prepend constituents " install_includes
"
18
A minimal Atlas package (1/3)

• gt cd mydevarea
• gt cmt create A A-00-00-01
• gt cd A/A-00-00-01
• gt echo " " gt ChangeLog
• gt mkdir A doc
• gt cvs import m "creation" offline/A Atlas
  A-00-00-01
• gt cd mydevarea
• gt cmt remove A A-00-00-01

• Creation
• CVS re-tagging
• Need an authorizing entry in tag availability CVS
  configuration file
• Registration for nightly and releases
• Need agreement from release coordination
• Use the tag collector to add (first) the package
  and to declare its tag

creation
gt cd mydevarea gt cmt co A gt cd A/A-00-00-01/cmt gt
vi requirements Use AtlasPolicy AtlasPolicy- ...
modifications
19
A minimal Atlas package (2/3)

• Minimal package structure
• Package naming convention
• XxxAlgs Athena algorithms
• XxxSvc Athena Service
• XxxUtils Utility classes
• XxxEvent Event data
• XxxDetDescr Detector description data
• XxxConditions Conditions data

cmt
/
requirements
/
.cxx
/
src
/
subsystem
A
A-00-00-01
/
/
A
/
.h
/
README mainpage.h
/
doc
/
20
A minimal Atlas package (3/3)

• Links
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• architecture/General/Documentation/PackageStruct
  ure.txt

21
Building libraries and applications (1/6)

• Building a normal (standard) shared library
• Sources should be by default in ../src
• Generally library name should be the package name
• A library is automatically installed in the
  installation area, thus the linker option only
  needs lltnamegt since the L specification will be
  automatically specified towards the installation
  areas (by CMT)
• LD_LIBRARY_PATH should not be specified (for the
  same reason as for linkopts)

package A author Mister.A_at_cern.ch use
AtlasPolicy AtlasPolicy- library A .cxx macro
A_linkopts " lA "  
Required for all packages
Specification for clients of A
22
Building libraries and applications (2/6)

• Building a normal application
• Sources should be by default in ../src
• An application is automatically installed in the
  installation area
• PATH should not be specified (it is automatically
  set to the installation area by CMT)

package B author Mister.B_at_cern.ch use
AtlasPolicy AtlasPolicy- use A
A-00- application B .cxx
Only the major id is specified, all other version
specs are wildcarded
23
Building libraries and applications (3/6)

• Building an athena component
• The package name should reflect the kind of
  component(s)
• XxxAlgs
• XxxSvc
• Structure

cmt
/
src
/
subsystem
A
A-00-00-01
/
/
A_load.cxx A_entries.cxx
src
/
components
/
/
A
/
doc
/
share
/
Joboptions Data files ...
/
24
Building libraries and applications (4/6)

• Athena component files
• XxxAlgs_entries.cxx
• XxxAlgs_load.cxx

include "XxxAlgs/MyAlgorithm1.h" include
"XxxAlgs/MyAlgorithm2.h" include
"XxxAlgs/MyAlgTool.h"   include
"GaudiKernel/DeclareFactoryEntries.h" DECLARE_ALGO
RITHM_FACTORY( MyAlgorithm1 ) DECLARE_ALGORITHM_F
ACTORY( MyAlgorithm2 ) DECLARE_TOOL_FACTORY(
MyAlgTool ) DECLARE_FACTORY_ENTRIES( XxxAlgs )
DECLARE_ALGORITHM( MyAlgorithm1 )
DECLARE_ALGORITHM( MyAlgorithm2 )
DECLARE_TOOL( MyAlgTool )
On entry per algorithm And per tool
On entry per algorithm And per tool
include "GaudiKernel/LoadFactoryEntries.h" LOAD_F
ACTORY_ENTRIES( XxxAlgs )
25
Building libraries and applications (5/6)

• The requirements file
• If the library is a pure component
• Or, if the library can also be used as a normal
  shared library

package XxxAlgs author Mister.XxxAlgs_at_cern.ch us
e AtlasPolicy AtlasPolicy-01- use
GaudiInterface GaudiInterface-01- External
library AthExHelloWorld .cxx -scomponents
.cxx apply_pattern component_library
... apply_pattern dual_use_library
26
Building libraries and applications (6/6)

• Links
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• architecture/General/Documentation/DualUseLibrar
  ies.txt
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• architecture/General/Documentation/PackageStruct
  ure.txt
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• architecture/General/Documentation/StandardPatte
  rns.txt
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• architecture/General/Documentation/AthenaDevelope
  rGuide.pdf

27
Building test programs (1/2)

• Conventions, policies and tools are all
  experimental and will surely evolve
• The athenarun_test pattern
• Available from TestPolicy
• Launch athena.exe against one joboption file
• Produces one log file
• The ctest pattern
• Allows to test the package components
  individually.
• One C source file
• One header file
• The component test source.
• Convenient for testing simple classes and
  associated functions, Athena services and
  algorithms
• http//www.usatlas.bnl.gov/computing/software/comp
  onent_testing.html

apply_pattern athenarun_test name"RecExCommonTest
ing" \ pre_script"../Testing/script_pre_check.s
h" \ options"RecExCommon_jobOptions.txt" \
post_script"../Testing/script_post_check.sh"
28
Building test programs (2/2)

• Links
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  sit/Testing/
• http//www.usatlas.bnl.gov/computing/software/comp
  onent_testing.html
• http//atlas-sw.cern.ch/cgi-bin/cvsweb.cgi/checko
  ut/offline/
• AtlasDoc/doc/sit/UserDev_HowTo.htmlrun
  automatic test

29
The documentation generator (1/3)

• Make use of doxygen
• Automatically acquired through AtlasPolicy
• May be manually run on any package
• gt gmake doxygen
• Can be customized
• All customisation macros are named DOXYGEN_xxx
• The list of customisable parameters is limited
  but could be easily extended on demand
• Produce automatic links (URLs) to symbols defined
  in used packages

cmt show macros DOXYGEN_ DOXYGEN_INPUT'../src
../(package) ../doc' DOXYGEN_FILE_PATTERNS'.cxx
.h' DOXYGEN_EXAMPLE_PATH'../doc
../cmt' DOXYGEN_EXAMPLE_PATTERNS'.cxx .html
requirements' DOXYGEN_SORT_MEMBER_DOCS'NO' DOXYGE
N_INLINE_INHERITED_MEMB'NO'
30
The documentation generator (2/3)

• A convention is to provide a ../doc/mainpage.h
  header file containing doxygen statements in one
  C comment section.
• The resulting doc is installed in
• CMTINSTALLAREA/doc/ltpackagegt/html
• A web page is available for accessing the doxygen
  output for the releases
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• Release/Doxygen

This Doxygen keyword means that this document is
the mainpage
/ _at_mainpage This package ... / /
_at_example example1.cxx first example. /
Install the code of an example
31
The documentation generator (3/3)

• Links
• http//atlas-sw.cern.ch/
• cgi-bin/cvsweb.cgi/checkout/offline/
• AtlasDoc/doc/sit/UserDev_HowTo.htmldoxygen
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  
• Development/qa/Documents/DoxygenComments.html

32
Quality checking tools (1/2)

• The C coding style checker tool
• Make use of the RuleChecker utility
• Automatically acquired through AtlasPolicy
• May be run on any package
• gt gmake rulechecker
• The output is sent to ../CodeCheck/rulechecker/.v
  iol
• Links
• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/
  Development/
• qa/General/index.html

33
Quality checking tools (1/2)

• The checkreq.sh script
• Compare the nominal package dependencies (as
  determined by the "use" statements) with the
  needed dependencies (as determined by C
  "include" statements and possibly additional
  libraries that are required)
• Daily applied in the nightly. Results are
  displayed in the nightly page
• Should help simplifying and rationalizing the
  dependencies
• Its a good indication to the truth
• Current Atlas policy
• Always explicitly use AtlasPolicy
• Always explicitly use packages corresponding to
  include statements (ie transitive dependencies
  should not be assumed)
• Can be interactively applied from the cmt
  directory
• gt checkreq.sh -help -w
• Shows apparently unneeded packages
• Shows missing use statements wrt C include
  statements
• Can generate the basic use statements (option w)
• http//atlas-sw.cern.ch/cgi-bin/cvsweb.cgi/checko
  ut/offline/AtlasDoc/
• doc/sit/UserDev_HowTo.html

34
Installing files (1/4)

• Libraries and applications are always
  automatically installed in the installation area.
• CMTINSTALLAREA/i686-rh73-gcc32-dbg/lib/lib.a
• /lib.so
• CMTINSTALLAREA/i686-rh73-gcc32-dbg/bin/.exe
• Applicable when CMTINSTALLAREA is defined
• Automatically defined WRT the first item in
  CMTPATH
• ltdev-areagtltprod-areagtltgaudi-areagt
• If current directory is outside the first item of
  CMTPATH, installation does not happen.
• Eg when ltdev-areagt is not specified in CMTPATH

35
Installing files (2/4)

• Include files are also automatically installed
  (before all other constituents are built)
• CMTINSTALLAREA/include/ltpackagegt/ltpackagegt/.h
• Two levels of ltpackagegt directories are
  constructed to force one I option per package
  (rather than one option per installation area)
• -ICMTINSTALLAREA/include/ltpackagegt
• Standard include files (../ltpackagegt) are
  automatic
• Non standard include directories must be
  explicitly declared using the install_non_standard
  _include pattern
• One apply_pattern per non standard include
  directory
• Non standard directories are simlinked into
  CMTINSTALLAREA/include/ltpackagegt/aaaa
• gt generally the standard I option will support
  them

apply_pattern install_non_standard_include
nameaaaa apply_pattern install_non_standard_inclu
de namebbbb
36
Installing files (3/4)

• All other files need to be explicitly declared to
  have them installed.
• Files are expected from ../share in the package
• Except for declare_xmls ../data
• A standard macro contains the list of files
• declare_joboptions and declare_xmls both add a
  ltpackagegt directory level

file type pattern destination param default path macro
jobOptions declare_joboptions joboptions files None JOBOPTSEARCHPATH ltpackagegt_joboptions
Data files declare_runtime share files None DATAPATH ltpackagegt_runtime
Scripts declare_scripts share/bin files None PATH ltpackagegt_scripts
XML files declare_xmls XML extras .xml .dtd XMLPATH ltpackagegt_xmls
apply_pattern declare joboptions files".txt
.py" apply_pattern declare_runtime
files"a.data  apply_pattern declare_xmls
37
Installing files (4/4)

• Installing a file into the work area
• Needed for software which require the datafiles
  in the current directory
• Also needed for modifying runtime files
• Eg Joboptions
• From an interactive shell session Use a script
• gt get_joboptions ltlistgt
• Supports auto-completion mechanism of the shells
  (zsh/tcsh)
• gt get_joboptions Alttabgt gt AthExHelloWorld/HelloW
  orldOptions.txt
• Immediately copies the joboption file(s) into the
  current directory from the installation area
• Generic version
• Also supports the auto-completion mechanisms
• gt get_files jo ltlistgt -data ltlistgt -scripts
  ltlistgt -xmls ltlistgt
• From the TestRelease package use a pattern
• apply_pattern get_files joltlistgt dataltlistgt
  scriptsltlistgt xmlsltlistgt
• Applied when running gmake get_files

38
Setting up for the distribution kit (1/2)

• Internal packages
• Basically nothing special is required. The
  following files are automatically included in the
  kit
• All libraries and applications
• Standard include files (ie ../ltpackagegt)
• Non standard include files that are declared
  using the install_non_standard_include pattern
• Runtime files that are declared using the
  declare_xxx pattern
• The tar ball contains two sections
• The CMT files (requirements, make fragments, )
• Replicate the complete package directory
  organization
• Permit the development of user packages against
  the kit
• The installed files
• Contain the subset of the installation area
  related with the package

dist/7.2.0/Control/AthenaExamples/AthExHelloWorld/
AthExHelloWorld-00-01-06/cmt/requirements dist/7.2
.0/InstallArea/i686-rh73-gcc32-dbg/lib/libAthExHel
loWorld.so dist/7.2.0/InstallArea/joboptions/AthEx
HelloWorld/HelloWorldOptions.txt dist/7.2.0/Instal
lArea/joboptions/AthExHelloWorld/HelloWorldOptions
.py dist/7.2.0/InstallArea/include/AthExHelloWorld
/config.h dist/7.2.0/InstallArea/include/AthExHell
oWorld/HelloWorld.h dist/7.2.0/InstallArea/include
/AthExHelloWorld/config.h.in
39
Setting up for the distribution kit (2/2)

• External packages
• One special CMT macro is required to assemble the
  tar ball containing the physical stuff.
• ltpackagegt_export_paths
• Specifies a list of file paths (directories or
  files) that will be accumulated into the tar ball
• All items in this list must be specified so that
• If at CERN (ie when the CERN tag from CMTSITE -
  is true) all items points to the actual stuff
• When the STANDALONE tag is true all items should
  be defined relatively to SITEROOT
• Eg

XercesC_export_paths'(XercesC_install)/include
\ (XercesC_install)/lib/libxerces-c.(shlibsu
ffix) XercesC_install'(XercesC_home)/(XercesC
_native_version)/(XercesC_system)' XercesC_home'
LCG_EXTERNAL/XercesC' LCG_EXTERNAL'LCG_ROOT
/external' LCG_ROOT'SITEROOT/sw/lcg'
40
Discussion

• Questions?
• Comments?
• HowTos?