Title: ETICS The Software Engineering Infrastructure
1ETICSThe Software Engineering Infrastructure
- Physics Services Meeting
- Geneva, July 3rd 2008
Alberto Di Meglio CERN - ETICS alberto.di.meglio_at_c
ern.ch
2Contents
- General overview
- Architecture
- What ETICS is and what it is not
- System features highlights
- Guidelines and constraints
- Benefits of using ETICS
- Problems and Issues
- Examples of usage
- Conclusions
3General Overview
- ETICS stands for
- E-infrastructure for Testing, Integration and
Configuration of Software - ETICS started in January 2006 and ended in
December 2007. It had a very successful review
exceeding all stated objectives - ETICS 2 started in March 2008 (ranked first in
the proposal selection process) and its supposed
to run until February 2010 (mostly with EC
funding) and then on its own - ETICS is not just a build system, its a
complete infrastructure for building, testing,
configuring and managing software projects
4Partners
5ArchitectureETICS is not just a build system
6What ETICS is
- Its a software engineering management system
- Its a build and test infrastructure
- It provides tools and resources to configure,
manage and analyse build and test runs - It provides a common interface to diverse
projects to facilitate knowledge sharing and
operations management - It has an open repository of configuration
metadata, packages, reports. The goal is to share
information, but also to reliably store and
preserve information - It has a plugin-based architecture and APIs to
allow integrating ETICS into existing processes
and extending it with custom actions - Its multi-platform and independent from any
specific build or test tool
7What ETICS is not
- Its not a replacement for source code management
systems like CVS or Subversion. ETICS uses such
systems and can be easily extended with support
for more - Its not a replacement for build tools like make,
ant, etc. ETICS uses whatever native tool a
specific project decides to use and doesnt force
the usage of any particular tool - Its not a replacement for QA tools like
checkstyle, junit, cppunit, coverage tools, etc.
ETICS provides a rich library of such tools that
projects can activate as they wish when running
builds and tests
8Typical ETICS Execution Sequence
etics-get-project
etics-checkout
etics-build/test
Extract metadata information about a project from
the ETICS DB
Extract configuration information from the ETICS
DB and source/binary packages from different
repositories
Execute the build or test operations
make
ant
cvs
svn
http
Unit tests, coverage, service mgmt, packaging,
reporting
9Plugins and Metrics Collectors
- The ETICS system is plugin-based (like for
example ECLIPSE) - It provides a core set of tools and a published
specification for developing additional plugins - Plugins are essentially thin wrappers around
existing or custom tools providing very specific
functionality like packaging, static or dynamic
testing, standards compliance testing, service
installation and management, reporting, etc - Plugins can publish information in the ETICS
system in the form of metrics, which can then be
used to do data mining or trend analysis using
the available ETICS reporting tools
10Examples of metrics collectors
11Examples of metrics collectors
12The Distributed Testing Feature
- One of the last features to be added, still in
experimental mode - It allows designing complex tests involving
several services and test applications to be
deployed on separate nodes - ETICS analyses the definition and deploys the
services on the necessary nodes - A synchronization mechanism orchestrate the
start/stop of services and the execution of the
test applications - At the end the results are collected and reported
as a single report - It is not yet usable by any user, it requires
some deep knowledge of the system to be tested - It has been successfully used in a number of
cases by the DILIGENT project - It has the strong prerequisite that the services
to be deployed have to be managed without user
intervention, which is not always the case
13The ETICS Infrastructure
- The ETICS Infrastructure is currently based on
three resource pools managed by Condor, at CERN,
UoW and INFN - There are about 450 cores available in the three
sites with more than 40 different types of
platforms - At CERN in particular there are 120 cores and 12
different platforms (a platform is a combination
of operating system, CPU architecture and
compiler, ex slc4_ia32_gcc36, deb4_x86_64_gcc412,
etc) - Additionally various nodes are attached to the
CERN pool from third-party projects or
organizations for specific use (4D Soft Ltd for
DILIGENT in Hungary, GARR for EUChinaGrid and
EGEE/SA2, IN2P3 for EGEE/SA2)
14The CERN Resource Pool
15Guidelines and Constraints
- The ETICS System is quite flexible in terms of
usage policies - However, there are recommendations and
constraints - The most important are
- Organize the software in well manageable units
(components) and functional sets (subsystems).
This is not mandatory, projects are free to
organize the software as they wish - Version the software when changes occur to be
able to reproduce past configurations and
generate reliable trend analysis data. Not
mandatory, but there is a lot to lose in not
doing it
16Guidelines and Constraints (2)
- Lock the software when its ready to be released,
so it cannot be modified anymore, and give it a
new name/version/date. This is not mandatory,
however it is strongly recommended and related to
the next point - Only locked configurations can be published in
the one and only public permanent Repository.
This is a hard constraint. However, users can
create as many volatile repositories as they wish
during the development and test phases
17Benefits of Using ETICS
- Common set of interfaces and tools to different
projects - Simplifies working with different projects and
transferring knowledge among technical people - Configuration information is stored in a single
place with a common schema and preserved while
people or projects change - Multi-platform support allow building and testing
the same code-base on as many platforms as needed
with negligible additional effort - The resource infrastructure allows running
several builds and tests without having to set up
dedicated machines - All machines are configured in the same way or
use virtual images in order to guarantee
reproducibility of results - Standard build and test tools are available
out-of-the-box and can be applied in the same way
to different build/test runs and different
projects over time
18Benefits of Using ETICS (2)
- The built-in packaging system abstracts the
package information and automatically builds the
appropriate package format for the different
platforms (tarballs, RPMS, debs, MSIs, etc). No
need to maintain separate scripts - Reporting tools are available out-of-the-box to
control the software evolution, pinpoint issues
and quickly solve problems - APIs and plugins allow to integrate the tools
into existing development processes and extend
them with custom tests or reports as needed - New plugins and tools developed by a project can
be of benefit to all other projects (no
duplication of effort) - ETICS provides a large set of external libraries
in source format or already precompiled for many
platforms (the externals project). Components
can set dependencies on these libraries and
automatically configure their workspace.
19The externals project
20Problems and IssuesTechnical problems
- PERFORMANCE
- The system was designed for integrators and
managers and the speed of execution of individual
commands was not a priority compared to support
for multiple platforms, reporting, common
interfaces - Over time its been used more and more by
individual developers, whose primary concern is
performance of single builds or tests, rather
than quality evolution over time - The original XML-based implementation did not
scale, new implementation is based on sqlite, the
de-facto standard in multiplatform embedded
database engines - New requirements have been analysed and a new
version of the tools is being deployed that
improves performance from 200 to 900 depending
on the task to be executed and the available
hardware
21Performance ComparisonsCheckout and workspace
configuration
- The checkout and workspace configuration
(etics-checkout) is the command with the highest
overhead. The table above compares the execution
time for the entire gLite project and some
individual subsystems - The commands are executed on a standard CERN
desktop with SLC4 32 bit. It may take longer on a
laptop or older machines (it requires at least
1GB RAM to handle the entire gLite project) and
from outside CERN (due to CVS)
22Problems and IssuesTechnical problems
- USER-FRIENDLYNESS
- The system has some learning curve, although it
depends on the user background and experience - In part this is due to the fact that the scope of
the system is quite broad and there are many
different commands and options to be used - But it is true that the interfaces (both web and
CLI) could benefit from a number of techniques to
make their usage friendlier (wizards, templates,
etc) - User documentation is extensive, but not in a
format that developers like to use. We are moving
now from a single Word document to lightweight
web-based help pages and tutorials - It must be said that once any person learns how
to use the system, the same knowledge can be
applied to any of the registered projects, making
sharing of effort within teams very easy
23Problems and IssuesSocio-technical problems
- Projects that started using ETICS since the very
early phases had to endure the unavoidable
initial problems and bugs. In many cases this had
created a certain mistrust in the system that we
have now difficulties in overcoming - Many developers are very reluctant to use new
procedures and tools, especially when they have
developed their own. We still have several users
wrapping the ETICS tools within layers of
scripts. These scripts are supposed to improve
the tools and/or shield the developers against
their misbehaviour, but often prevent them from
working correctly - In other cases, projects use their own tools or
store some information outside the system in
parallel to ETICS. This prevents ETICS from
providing some functionality for lack of complete
information. Although this in itself is not a
problem, it becomes a problems if it is perceived
as an issue on the ETICS side
24Problems and IssuesSocio-technical problems
- Developers independence is important to keep up
motivation and enthusiasm. However for the
managers is good to have some uniformity, global
reporting and trend analysis. These two aspects
are very difficult to reconcile - ETICS is configurable to allow a sort of
intermediate managed freedom, but some rules
have to be established
25Example of UsageEGEE
- EGEE uses ETICS within four Activities NA4,
JRA1, SA2, SA3 - JRA1 and SA3 are the main ETICS users (more in
the following slide) - SA2 has implemented with ETICS the IPv6
compliance analysis of gLite code and the
distributed IPv6 experimental testbed where the
IPv6 version of BDII has been tested - NA4 is using ETICS for experimenting with a few
projects like Gridway, mpi, SAGA and the panc
quattor compiler
26Example of UsageEGEE gLite Building
- org.glite 289 Modules, 5211 Configurations, 149
Main Reports, 1583 Metrics - The project has two main configurations, one for
production (glite_branch_3_1_0) and one for
development (glite_branch_3_1_0_dev) - The project configuration is not versioned,
although recently some backups have been created
when a change occurred - All other configurations are versions of the 288
modules inside the org.glite project - It is built four times per day on 5 different
platforms (slc4 32/64, sl5 32, deb4 32, rhel4
32). SLC3 has been dropped recently - Many on-demand builds are performed by individual
developers - The porting of several services is managed with
ETICS. At the moment development gLite builds are
run on several different platforms
27Examples of UsageEGEE gLite Porting
28Example of UsageEGEE gLite Testing
- Very little testing is done with ETICS at the
moment - There is ongoing work to do deployment and patch
testing of the gLite metapackages (services) - However, this is made difficult by the fact that
the definitions of the metapackages and patches
are not in ETICS - We are working on a Savannah plugin that would
allow extracting information from Savannah (bugs,
patches, etc) - More complex tests are planned but only after
this is solved
29Example of UsageDILIGENT/D4Science
- DILIGENT has used ETICS for building, testing,
releasing and reporting since the beginning - D4Science, its follow-up project, has fully
standardized its software engineering process on
ETICS - The are two main projects in D4Science
- org.gcore 14 Modules, 619 Configurations, 60
Main Reports, 160 Metrics - org.gcube 145 Modules, 1386 Configurations, 131
Main Reports, 272 Metrics - A letter has recently been received from the
D4Science Technical Coordinator describing how
they use ETICS and quantifying the savings it
brought in terms of manpower and efficiency of
the process
30Example of UsageOMII-Europe
- OMII-Europe has used ETICS as software
engineering platform since the beginning - It mainly includes builds of the BES compliant
modules for CREAM CE and UNICORE - BES compliance tests for CREAM-BES have also been
implemented in ETICS, although problems in
deploying CREAM have prevented running them in a
completely automated way - OMII-Europe 13 Modules, 10 Configurations, 10
Main Reports, 86 Metrics
31Experimental ProjectsROOT
- ROOT has been added to ETICS by ETICS developers,
not by ROOT developers. It has been used as a
proof of concept to see how difficult it is to
add existing projects to the system - It was added and configured in one afternoon
- It can be easily built with ETICS on various
platforms (slc3, slc4 32/64, rhel4, debian) - root-project 1 Module, 6 Configurations, 6 Main
Reports, 12 Metrics
32Experimental ProjectsCASTOR 2
- CASTOR 2 has been added to ETICS by ETICS
developers as a proof of concept to see how easy
it is to add existing projects to the system - The CASTOR developers have been involved in this
case - The exercise started nominally more than one year
ago with very low priority on both sides and a
bit more effort only in the past two weeks (also
in preparation of this presentation) - However, it still cannot be fully built due to
some compilation errors and a rather different
approach to building software - The errors can be reproduced with and without
ETICS and have been shown to the developers. Work
is ongoing.
33 34Next Steps
- The major goals of ETICS 2 are
- To make the system even more reliable and
user-friendly - To include it as a general service in the major
European infrastructures (EGEE, DEISA, SeeGrid,
etc). ETICS support is already a standard unit
within GGUS - To propose it also to non-grid related projects
inside and outside CERN - To add direct support for major job management
systems (CREAM, UNICORE, LSF). Currently only
Condor is supported - To add more advanced release management and patch
management tools with proper responsibility
transitions (like in EDH for example) - To add dedicated project dashboards where
information about projects is clearly summarized
(like in SourceForge)
35Conclusions
- ETICS is not a build system
- ETICS is a complete infrastructure for
configuring, building, testing and managing the
software development process - The current system is the result of two years of
requirements analysis, design, implementations
and deployment with many projects and developers - It is currently effectively used in production
tasks - Problems do exist, but we are actively and
proactively addressing them together with the
users - Where we go from here depends on how we can
evolve the system and how many more projects
decide to adopt ETICS