The Virtual Data Toolkit

1
The Virtual Data Toolkit OSG Software

• Alain Roy

OSG Software Coordinator
Condor Team
University of Wisconsin-Madison
2
Review What is the VDT?

• A collection of software
• Grid software (Condor, Globus and lots more)
• Virtual Data System (Origin of the name VDT)
• Utilities
• An easy installation
• Goal Push a button, everything just works
• Two methods
• Pacman installs and configures it all
• RPM installs some of the software, no
  configuration
• A support infrastructure

3
Who makes the VDT?

• The VDT is a product of Open Science Grid (OSG)
• VDT is used on all OSG grid sites
• OSG is new, but VDT has been around since 2002
• Originally, VDT was a product of the
  GriPhyN/iVDGL
• VDT was used on all Grid2003 sites

4
Who makes the VDT today?
1 Mastermind 3 FTEs
Miron Livny
Alain Roy
Expanding with new funding
Tim Cartwright
Andy Pavlo
5
Why have the VDT?

• Everyone could download the software from the
  providers
• But the VDT
• Figures out dependencies between software
• Works with providers for bug fixes
• Provides automatic configuration
• Packages it
• Tests everything on thirteen platforms (and
  growing)

6
Example VOMS

• VOMS can authorize people in a VO
• VOMS has a web interface
• We
• Install Tomcat
• Install Apache
• Built with Globus SSL
• Patched so GSI pass-through to Apache works
• Install VOMS
• Install VOMS Admin
• Install Perl modules needed by VOMS Admin
• Install MySQL and set up database (with
  command-line tool)
• Configure all software
• Configure rotation of log files

7
Example Security Update

• Last week, we got security update from Globus
• We decided to patch VDT 1.2.x, 1.3.9, 1.3.10, and
  1.3.11
• We need to build the updated binaries three times
  on about six platforms
• We need to coordinate creation of patch for
  Globus 2.4.3
• We need to patch the Globus updates with our
  patches
• We need to take subset of Globus updates
• We need to package an update that can be
  reversed, if there are problems.

8
How much software?
9
Current Release VDT 1.3.11

• Basis for OSG 0.5.0
• Adds some of requests for OSG 0.6
• Globus 4.0.2 with TeraGrid modifications
• SRM Clients
• Squid
• Wget
• New versions of CEMon and VOMS
• Support for Fedora Core 4 on x86-64

10
VDT 1.3.12 In Progress

• Likely to include
• Gratia (for accounting)
• BDII (including OpenLDAP)
• glexec
• Edge Services Framework
• TclGlobus
• Improvements to upgrading the VDT
• A slew of bug fixes and smaller enhancements
• More at http//vdt.cs.wisc.edu/future.html
• Final decisions about content during this meeting

11
VDT Stack gt OSG Stack
Input from stakeholders and OSG directors
VDT Release
OSG Integration Testbed Release
OSG Production Release
12
Making a VDT release

• Requires at least one week of testing on VDT
  testbed
• Run nightly tests on each supported platform
  until they are clean

13
Making Int. Testbed Release

• Add software (mostly config)
• Package subsets of VDT software
• Compute Element
• Storage Element
• Worker-node Client
• Submit-host Client
• Deploy on testbed
• Test services
• Test middleware interoperability
• Test functionality and scalability
• Application validation

14
Challenges we face

• How should we smoothly update a production
  service?
• In-place vs. on-the-side
• Preserve old configuration while making big
  changes.
• As easy as we try to make it, it still takes
  hours to fully install and set up from scratch
• How do we support more platforms?
• Its a struggle to keep up with the onslaught of
  Linux distributions
• AIX? Mac OS X? Solaris?

Fedora Core 6
Gentoo
Fedora Core 5
Fedora Core 4
RHEL 3
BCCD
Fedora Core 3
RHEL 4
Debian
RHEL 5
15
More challenges

• Distributed creation of VDT
• Now we have funding for people to add support for
  new storage resource management systems (d-Cache,
  DRM, etc)
• These people are not at UW-Madison
• Improving testing
• We care about interactions between the software
  When using a VOMS proxy with Condor-G, can we
  run a GT4 job with GridFTP transfer, keeping the
  proxy in MyProxy, while using PBS as the backend
  batch system
• Some people want native packaging formats
• RPM
• Deb

16
Where do you learn more?

• http//vdt.cs.wisc.edu
• Support
• Alain Roy roy_at_cs.wisc.edu
• Miron Livny miron_at_cs.wisc.edu
• Official Support vdt-support_at_ivdgl.org