SKAKAT SPIN Presentation

1
SKA/KAT SPIN Presentation

• Software Engineering (!?)
• 2007-01-17
• Robert Crida

2
Overview

• Software Configuration Management (SCM)
• Automated Build System
• External Dependencies
• Design Study Antenna Control Software Module
  (ACSM)

3
SCM Challenge

• KAT is expected to have a life span in excess of
  20 years (project will outlive the people)
• Maintenance is inevitable during that period
• Continuous development is also likely
• Importance over that period of being able to
  freeze build environment
• What about hardware failure that far into the
  future?

4
SCM

• Use subversion potentially multiple
  repositories (collaborations, different
  subsections)
• Store multiple projects in a tree
• Standard svn hierarchy for each project
• trunk (primary development)
• tags (releases)
• branches (qa, users, etc)
• Try to include all dependencies as projects
  within the hierarchy

5
Subversion Layout

• Tools (used by the build process)
• doxygen
• tags
• doxygen-1.4.7
• doxygen-1.5.1
• etc (typically only store tags for each tool)
• Code (ultimately linked into products)
• ThirdPartyLibraries (typically only tags)
• InternalProjectsHierarchy
• Each layer can only depend on other projects in
  the same or lower layers, ideally wanted to only
  depend on projects in lower levels but deemed too
  difficult for users, instead have to work out
  build order within each layer.

6
Python projects

• Build system based on distutils
• Actually we use from recursivebuild import setup
• recursivebuild.setup delegates to
  setuptools.setup and provides a convenient place
  to centrally add new commands (eg doc, pylint)
• Use setuptools because of egg support
• Allows multiple builds of the same library to be
  present simultaneously
• from pkg_resources import require
• require(numpy1.0)
• import numpy
• Most distutils projects easily converted by
  changing import of setup function

7
Build System

• Each directory of the hierarchy has a script to
  build its children
• script determines order
• passes arguments from parent
• arguments modeled after distutils.setup
• Leaves are the projects themselves, must provide
  script understanding build targets
• Python projects use setup.py
• Other projects provide eg build.py to wrap make
  or scons etc

8
Build system contd 1

• All scripts must understand standard build
  targets
• install (more like build)
• puts python eggs into suitable directory
• should compile and link other libs but not deploy
• test (runs unit tests on the project)
• doc (runs doxygen on project)
• pylint (runs pylint on project rename to lint!)

9
Build system contd 2

• Build controller machine stores svn repository
  mirror plus overall build script
• Build environment to be configured on a virtual
  machine, currently requires only gcc and python
• Virtual machine is cloned prior to each build
• svn repository mirror is updated and copied to
  the build vm
• Multiple build steps are executed remotely on the
  vm by the controller

10
Build system contd 3

• This allows multiple vm's with different compiler
  versions and even OS versions
• Each can be built using (probably) the same
  control script

11
Testing

• Integrate unit testing based on xUnit into the
  auto-build
• Investigating Fitnesse based testing for
  automated ATP type testing
• Looked at web testing technologies such as Twill
  and selenium to assist with automated web app
  testing

12
Dependencies

• Python projects can depend on other python
  projects using the egg mechanism, hence can just
  find suitable egg in standard PYTHONPATH
• Depend on other projects by specifying paths
  within the hierarchy
• multiple versions of a lib can coexist
• requires extensive include and lib paths
• use PROJECT_ROOT environment variable instead of
  relative paths due to svn moves (different
  nesting levels)

13
External Dependencies

• Try to minimize external dependencies
• all need to go into SCM
• increases code-base (potential maintenance
  problem)
• traded off against leveraging existing code
• AIPS
• many dependencies
• hard to build difficult to configure tools
• constantly changing
• Use each one's own build system
• must configure appropriate include and lib paths

14
External Python Dependencies

• Eggs help a lot in theory
• Despite distutils, many python packages have
  non-standard builds which makes life difficult
• Haven't yet found a python project which defaults
  to using setuptools