Handling of Unix Application Software

1
Handling of Unix Application Software
Stephan Wiesand DESY -DV - May 25, 2004
2
Motivation

• in early 2003, we looked for a replacement for
  our Unix software deployment scheme
• install software in /afs/ltcellgt/products
• create symlinks in /afs/cell/_at_sys/usr/local
• all systems have these symlinks
• /products -gt /afs/ltcellgt/products
• /usr/local -gt /afs/ltcellgt/_at_sys/usr/local
• the replacement was designed, implemented and
  deployed since, as time allowed (no dedicated
  resources), and is presented here

3
The Problem

• provide additional software on all client systems
• sw not included with the distribution, or
• needed in different or multiple versions, or
• built with different options (kerberized...)
• targets
• Linux, Solaris, any other potentially relevant
  platform
• desktops, farm nodes, (workgroup) servers,
  controls, notebooks
• many have disks too small to install all software
  locally
• some must function properly without network or
  other infrastructure

4
Additional Requirements

• automatic (un)installation, up/downgrade
• configurable per client (tests...)
• keep systems in a well defined, working state
• reproducible without backup
• verifyable
• decent dependency handling (shared libraries
  allowed)
• within add-on software and against the host
  system
• for every client, even if some software isn't
  local
• lightweight solution, don't invent a new
  framework
• use standard tools, unmodified

5
Solution

• use a standard package format
• and a package database separate from the host
  system's
• split packages into
• a main package that can be replaced by a single
  symlink
• into a central reference installation
• an optional default package (always installed
  locally)
• populating directories in PATH, MANPATH etc. with
  symlinks
• populate the reference installation from same
  packages that would be installed locally
• on the client, maintain the package database as
  though all packages were installed locally

6
Packager Choice RPM (all platforms)

• available and working on all potential targets
• has all required features
• versioned dependencies/conflicts, virtual
  packages
• automatic for shared library dependencies
• --justdb switch for updating the database only
• structured package building from specs is a plus
• build procedure is fully documented automatically
• powerful macro facility (one spec for all
  platforms)
• writing specs is becoming a common skill
• excellent documentation for beginners available

7
Closing the Dependency Chain

• using a separate RPM DB
• is possible and consistent across platforms
• doesn't interfere with any mechanism maintaining
  the client
• but leaves a dependency gap between system and
  add-ons
• solution glue packages, providing
• virtual packages (ABI version of shared
  libraries, ...)
• ghost files (/bin/sh, ...)
• safeguards in preuninstall script enforce
  --justdb
• verifying a glue package will check that the
  system installation keeps its promises

8
Glue Packages

• to build a glue package,
• define macros for the missing dependencies
• for virtual/real packages add Provides ltmacrogt
• for files, add ghost ltmacrogt in files section
• provide a verify script
• checking existence of files (test -f ltmacrogt)
• or using /bin/rpm -q --whatprovides ltmacrogt
• can have one or more (glue-devel, glue-legacy,
  ...)
• possible enhancement mirror package for system
  DB
• requiring everything the glue package provides
  vice versa

9
Filesystem Layout Packaging

• deliberately chose not to use /usr/local

/opt/ products/ bin/
perl-gt/opt/products/perl/5.8.2/bin/perl
pine-gt/opt/products/pine/4.58/bin/pine perl/
5.8.2/ bin/ perl
pine/ 4.58/ bin/ pine
perl-5.8.2-2.i586.rpm pine-4.58-1.i586.rpm perl-de
fault-5.8.2-2.i586.rpm pine-default-4.58-1.i586.rp
m
10
Packages Rules

• base packages
• contain /opt/products/ltnamegt/ltversiongt
• content
• nothing else
• can be replaced by a single symlink
• to ltreference installgt/ltnamegt/ltversiongt
• default packages
• contain links to files from base packages
• anywhere in /opt/products
• except in any directory owned by a base package
• must require their base package

11
Refinements

• what if we want /opt/products/bin/perl5.8.0 ?
• -gt alias packages (like default, but multiple
  versions ok)
• what if we want to add files in a base directory
  ?(additional perl modules, configuration files)
• -gt daughter packages
• allowed to write anywhere below parent's
  directory
• must require their parent
• names prefixed by ltparent_namegt_ltparent_versiongt-
• what if we want to maintain certain files by
  other means?
• (again, configuration files)
• -gt local packages (like default, but links point
  to local fs)

12
Shared Library Dependencies

• we do populate /opt/products/lib (and include)
• but use is discouraged
• and it's not searched by the dynamic loader by
  default
• instead, set runtime search path for shared libs
  at build time
• (link) ... -Wl,-rpath,/opt/products/ltnamegt/ltversio
  ngt/lib ...
• sometimes hard to do (improper use of libtool...)
• means we have to rebuild dependent packages more
  often
• also means rolling out new versions doesn't break
  anything
• needs a modified find-requires not processing
  shared libs in /opt/products (spec macro
  __find_requires)

13
Deployment

• manual/scripted installation is possible
• mkdir -p /opt/products/RPMDB
• rpm -p --verify --verbose .../glue-2.0-2.i586.rpm
• rpm --dbpath ... -i --justdb .../glue-2.0-2.i586.r
  pm
• rpm --dbpath ... -i .../pine-4.58-1.i586.rpm
• mkdir -p /opt/products/perl
• ln -s ltreferencegt/perl/5.8.2 /opt/products/perl
• rpm -i --justdb .../perl-5.8.2-2.i586.rpm
• rpm -i .../perl-default-5.8.2.i586.rpm
• rpm -i .../pine-default-4.58-1.i586.rpm
• possible and straightforward, but tedious...

14
Tools

• instead, implemented ppm
• perl wrapper calling rpm
• finds packages in a repository on central
  filesystem
• locatedb for locating package files
• RPM DB (created with -i --justdb) for
  dependencies, content
• accepts input like this

install justdb glue 2.0 2 local pine 4.58 1
default link perl 5.8.2 2 default,alias
15
ppm features

• accepts one or more flat file names as input
• processed sequentially
• conflicting/redundant input ok
• last one takes precedence
• -gt overrides, delegation (also has an include
  mechanism)
• there's a remove tag as well
• set of input files defines absolute final state
• not an incremental change
• final state is verified sanitized - if possible
• resolve dependencies automatically
• correct children of base packages installed as
  link ...

16
Status

• so far, we mostly built packages for software
  that needed a rebuild or were unavailable before
• and a few essential ones
• and their dependencies
• in full production on DESY Linux 5
• most important packages available, few missing
• in use on DL4 and Solaris 8 hosts to varying
  degrees
• DL4 now obsolete, Solaris actually started this
  year
• on both sites (independent of host management
  solutions)
• also in use on Linux Notebooks

17
Experience

• improved preservation and transfer of knowledge
  about building an application or library properly
• no diligence (installation notes...) required
• package builders have to go through a learning
  curve
• about one day for building a first package not
  yet built
• starting from a similar example
• once used to it, little overhead to other
  procedures
• at least any producing transparent, reproducible
  results
• rebuilds and small changes much faster and more
  reliable
• we're probably past break even