SPI Infrastructure for LCG Software Projects

1
SPI Infrastructure for LCG Software Projects
LCG Software Process Infrastructure

• Overview
• A.Aimar EP/SFT
• CERN

2
Project context of LCG SPI
LHC grid software applications (LHC experiments,
projects, etc)

• LCG Application Area software projects
• POOL Persistency
• SEAL Core common software
• PI Physics Interfaces
• Simulation
• Detector Description
• etc

LCG Infrastructure

• Common services
• Similar ways of working (process)
• Tools, templates, training
• General QA, tests, integration, release

LCG SPI project
3
Project context of the LCG SPI

• Software Management Process RTAG
• General recommendations
• All LCG projects must adopt the same set of
  tools, standards and procedures
• Adopt commonly used open-source or commercial
  software where available
• Avoid do it yourself solutions
• Avoid commercial software that may give licensing
  problems
• If each project needs an infrastructure, many
  projects need it even more
• Tools, standards and procedures
• Try to avoid complexity

4
Infrastructure Software Development
Software development Support

• Keep it simple and modular approach
• a. Provide general services needed by each
  project
• CVS repository, Web Site, Software Library
• Mailing Lists, Bug Reports, Collaborative
  Facilities
• b. Provide solutions specific to the software
  phases
• Tools, Templates, Training, Examples, etc.

5
Project guidelines

• No specific software development
• The project should avoid any new development, and
  future maintenance, of any specific software
  package needed for the LCG software
  infrastructure.
• Use HEP or free software
• All software used should be already available, or
  become available, in the HEP community or in the
  free software community

• The goal of the project is to define an
  infrastructure and later a simple process
• Future maintenance will need separate planning
  and resources.
• Everything is done in collaboration
• LCG and LCG projects (Pool)
• IT division and LHC experiments
• Big projects (G4, Root, etc)
• Using as the existing IT services
• The project should use all existing IT services
• Make sure they match the LCG needs

6
http//spi.cern.ch
7
(No Transcript)
8
OUTLINE SPI Services

• Services specific to LCG projects
• General overview
• Very useful to LCG project but not really to
  other projects
• Services for LCG projects but available
• External software
• Testing framework
• General service for LCG and others
• Project Portal

9
gtgtgt CVS repository and Delivery Areas

• CVS repository
• A central CVS repository managed and available
  to all projects
• Any project just needs to ask for it, and declare
  its users permissions
• Managing mirroring and backups
• Users access controlled
• Tools for automatic clean up of locks, etc
• Systems to browse the repositories

• Delivery areas
• The AFS and DFS delivery areas
• an area to install software created by projects
  in the LCG application area
• an area where external and third party software
• an area where software is installed which is
  provided by people for evaluation within a
  project
• Pre-installed software shall be in directories
  specifying the package name, the package version
  and the OS_Compiler

10
(No Transcript)
11
gtgtgt Coding guidelines
Rule Checker (on the way) Human Reviews from SPI
12
gtgtgt CVS organization
13
gtgtgtScram

• Used by Pool and Seal
• Improvements
• restructuring the build system to improve
  performance
• possible to use automake/autoconf within a scram
  project
• Porting to Windows
• Execution on perl on Windows and usage of nmake
  (standard Windows installation without cygwin)
• Import projects into Visual Studio
• Organization
• a single SPI-managed toolbox service for LCG
  projects
• Organizing a tutorial and training for LCG
  developers and users

14
gtgtgt LCG Workbook
http//lcgapp.cern.ch/project/workbook/
15
Other components available

• Memory Leaks
• Automatic Builds (on the way)
• Software distribution (investigating pacman)
• Software documentation, templates, etc

16
gtgtgt Code Documentation

• Features of interest
• Code browsing
• Code searching
• Code information
• Design diagrams
• Any project will have them available as part of
  the infrastructure
• Doxygen
• LXRViewCVS

17
Code documentation Doxygen
18
Code documentation LXR
19
Code documentation ViewCVS

• CVSgraph Displays the tree of revisions and
  branches graphically
• Enscript Colorize files in the CVS repository.

20
gtgtgt External Software Service
21
Service Software Library

• All software needed by more than one LCG projects
  shall be installed in common areas
• Available for several compilers and operating
  systems
• Maintained centrally with support and help to the
  user
• External documentation available
• Contact with the authors if the software is
  developed in HEP or LCG
• Distribution and pre-installation to all LCG and
  HEP users

22
(No Transcript)
23
Current status

• Central installation of HEP or open source
  software needed by LCG related projects or
  following their requests in term of version and
  platforms (linux, MSWindows and Solaris when
  needed).
• Standard installation location
• Availability
• AFS /afs/cern.ch/sw/lcg/external/
• DFS \\cern.ch\dfs\project\lcg\external\ (just
  started)sundev005 /afs/cern.ch/user/l/lcgspi
  87gt ls /afs/cern.ch/sw/lcg/external/Ant/
  JACE/ Python/
  edg-rls-client/ png-lib/
  valgrind/Boost/ JDK/ TJDO/
  expat/ root/
  wxPython/CMake/ JDORI/
  XalanJ/ gccxml/ scripts/Colt/
  Log4J/ XercesC/
  mysql/ tarFiles/CppUnit/
  MMMySQL/ XercesJ/ mysql/
  test.htmlGSL/ Oval/
  doxygen/ packages/ uuid/

24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
gtgtgt Software Testing

• Tests of classes
• individual class
• small groups of
• collaborating classes
• work package

• Tools
• CppUnit
• PyUnit
• Oval

29
(No Transcript)
30
Test frameworks CppUnit
Similar tools Junit, PerlUnit, PyUnit,
QtUnit Output in XML, compiler or text Windows
version for MVC6.0

• A simple test
• Subclass the TestCase CppUnit class
• Override the method runTest().
• call CPPUNIT_ASSERT

31
Test frameworks CppUnit
32
Test frameworks Oval
Oval build
Test Source Code
Executable

• Oval
• validation
• regression

Oval run
Log file
Can be use for Unit-test It is possible to set
different run environments Can run external
scripts and external binaries Authors modified
it to run it with or with any environment
Reference file
Oval diff
33
Test frameworks Oval
34
Test structure and documentation
SPI-CVS_structure component

• CVS testing structure
• Test-naming

?Will help to have a automatic way to run test
code
SPI-TESTING component

• Test documentation
• Test cases specification template

35
Testing

• Available CppUnit, PyUnit and Oval
• Can also be used stand alone, integrated with
  Scram and with CMT
• Can also be used by projects NOT using Scram

36
gtgtgt Project Portal

• Project Portal
• The development front-end of every project
• With all features needed, configured and
  integrated
• News and Faq
• Task management
• Mailing lists access
• Bug tracking access
• Download area
• Access to the CVS repository
• We have customized Savannah from GNU

• Bug Tracker
• A central installation so that each project has
  an available bug tracker system
• Centrally managed but customizable for different
  needs
• Easier to post bug if projects use similar
  systems
• No work of maintenance and upgrade on the
  projects
• Web based
• http//lcgappdev.cern.ch
• (for a few test projects)

37
(No Transcript)
38
Benefits

• Totally web based
• Single entry point to project(s)
• Uniform access to project information
• Set up common web infrastructure for a project
  without coding

39
(No Transcript)
40
Bug tracker

• Fields customizable for each project
• File uploads
• Audit trail
• Full text searches
• Cc forwarding

41
(No Transcript)
42
News system

• Messages can be displayed in threads
• Monitoring (automatic sending of email)
• Link detection
• Transforms URLs to links
• References to bugs/tasks get linked

43
(No Transcript)
44
Task manager

• Sub-project/task hierarchy
• Tasks can be assigned to developers
• Tasks carry a deadline, etc.
• Tasks/bugs interdependencies

45
(No Transcript)
46
Personalize web page

• A logged in user gets
• a customizable personal entry page
• display of current work status
• User's bugs, tasks, ...
• Monitored news threads
• Bookmarks

47
(No Transcript)
48
(No Transcript)
49
And much more

• Consolidate project links
• ViewCVS (, Doxygen, LXR, Bonzai, ...)
• Project homepage
• Mailing lists (subscribe/unsubscribe)
• FAQ system (user/admin documents)
• Support manager
• Download area
• Patch manager
• Projects administrators can choose what to use
  for each project

50
Admin Project types

• Name and description
• Preferences common to one type of projects
• Homepage, CVS information
• List of status descriptions (e.g. alpha, beta,
  stable)
• Download/upload area (faq) information
• Allow use of other subsystems (patch manager,
  task manager, news, support request manager, bug
  tracker, FAQ)

51
Admin For each project

• Public information
• Name, description, development status
• Enable subsystems (bug tracker, patch manager,
  task manager, CVS link, news system, support
  tracker, FAQ system, download/upload area)
• Members
• Task manager, patch manager, support
  managernone, tech, admin techadmin
• Project administrator

52
(No Transcript)
53
Admin Bug tracking configuration

• Bugs
• Fields, values
• Email notification
• Global List of 'external' email addresses who
  get all new bug submissions
• Personal (techsadmins) List of 'Users to watch'
  
• Personal(techsadmins) Detailed list of events
  that trigger email to you
• News
• Delete postings

54
(No Transcript)
55
(No Transcript)
56
Specific implementations

• General bug fixing and improvements
• Implemented bulk user registration
• Get a list of users and add/remove them all
  without registering them one by one
• Integration with AFS authentication
• Added a PAM module to verify a login against the
  AFS authentication
• The user can still define a password not using
  the AFS password
• Sending back to Savannah our changes, patches and
  fixes
• Run a stable service
• http//lcgappdev.cern.ch for now

57
Proposed release schedule

• December 2002 - Beta Versions
• Many components will be available before
• February 2003 - Version 1.0
• All components and services mentioned above
  except
• Software Library (but basic), Project portal
  (but basic)
• Monitoring if resources will need to be injected
• May 2003 - LCG Test - Version 1.1
• Software Library, Project portal and all features
• Standard templates and tools available
• September 2003 - Version 1.2
• LCG software process
• Standards for user specifications
• Design guidelines and specifications
• end of 2003 until 2005
• Development Version 2, support and maintenance

58
Conclusion

• Homogeneity is important for the LCG and for its
  projects
• All work is done seeing the experts and work in
  LHC experiments and LCG projects
• We all need to compromise and NOT focus on
  infrastructure and individual taste
• The reference URL is http//spi.cern.ch the
  others in this presentation may change.

59
http//spi.cern.ch