Title: The Virtual Data Toolkit VDT
1The Virtual Data Toolkit (VDT)
- Miron Livny
- University of Wisconsin - Madison
- miron_at_cs.wisc.edu
- First EGEE Conference
- April 18-22, 2004
2VDTs Mission
- GriPhyN a primary GriPhyN deliverable will be
a suite of virtual data services and virtual data
tools designed to support a wide range of
applications. The development of this Virtual
Data Toolkit (VDT) will enable the real-life
experimentation needed to evaluate GriPhyN
technologies. The VDT will also serve as a
primary technology transfer mechanism to the four
physics experiments and to the broader scientific
community. - iVDGL VDT Robustification and Troubleshooting
team (VRT) .... will not develop new software but
instead will work to enhance the releases of the
GriPhyN VDT software, as follows - (1) develop and maintain a VDT test suite
- (2) define and implement a unified and fully
integrated error reporting framework across all
VDT components - (3) equip the VDT with dynamically configurable
event logging capabilities - (4) extend the VDT with new components required
for specific application purposes - (5) help users to maintain and trouble shoot the
VDT software, - (6) provide documentation, and
- (7) create procedures and tools for reporting,
tracking, and fixing bugs.
3What is the VDT?
- A collection of software
- Common Grid middleware
- Virtual data software
- Utilities
- An easy installation mechanism
- Goal Push a button, everything you need to be a
consumer or provider of Grid resources just works - Two methods
- Pacman installs and configures it all
- RPM installs some of the software, no
configuration - A support infrastructure
- Coordinate bug fixing
- Help desk
- Understand community needs and wishes
4What is the VDT?
- A highly successful collaborative effort
- VDT Working Group
- VDS (Chimera/Pegasus) team
- Provides the V in VDT
- Condor Team
- Globus Alliance
- NMI Built and Test team
- EDG/LCG
- Middleware, testing, patches, feedback
- PPDG
- Hardening and testing
- Pacman
- Provides easy installation capability
- Currently Pacman 2, moving to Pacman 3 soon
5VDT Timeline Highlights
- Fall 2001 VDT started by GriPhyN and iVDGL
- Supported US-CMS testbed in 2002
- March 2002 VDT support system inaugurated
- Early 2003 Adopted by European Data Grid LHC
Computing Grid - April 2003 VDT Testers group started
- Fall 2003 Supporting Grid3
- Fall 2003 Adopted by Particle Physics Data Grid
- Nov 2003 Nightly test infrastructure deployed
6VDT Growth
VDT 1.1.8 First real use by LCG
VDT 1.1.11 Grid2003
VDT 1.0 Globus 2.0b Condor 6.3.1
VDT 1.1.7 Switch to Globus 2.2
VDT 1.1.3, 1.1.4 1.1.5 pre-SC 2002
7Tools in the VDT 1.1.13
- Condor Group
- Condor/Condor-G
- DAGMan
- Fault Tolerant Shell
- ClassAds
- Globus Alliance
- Job submission (GRAM)
- Information service (MDS)
- Data transfer (GridFTP)
- Replica Location (RLS)
- EDG LCG
- Make Gridmap
- Certificate Revocation List Updater
- Glue Schema/Info prov.
- ISI UC
- Chimera Pegasus
- NCSA
- MyProxy
- GSI OpenSSH
- UberFTP
- LBL
- PyGlobus
- Netlogger
- Caltech
- MonaLisa
- VDT
- VDT System Profiler
- Configuration software
- Others
- KX509 (U. Mich.)
8The PPDG factor
- PPDG played a key role in the hardening of VDT
(US-CMS testbed) - PPDG experiments (BaBar, STAR, Run II, US LHC,
JLAB) have adopted the Virtual Data Toolkit
(VDT) packaging and distribution of the common
Grid middleware. - As part of the extension proposal (8/04-7/06),
PPDG will - - Align, to the extent possible, middleware
deployments and timelines with the testing and
release procedures of the VDT. - Contribute to testing of VDT by integrating
pre-release and development versions into their
test beds for end-to-end validation and system
tests. - Adopted services most likely to follow common
Web-Service based interfaces. The project will
work with the VDT on service-based components as
they emerge and evolve.
9Core VDT Working Group
- Miron Livny (lead)
- UW-Madison
- Alain Roy (technical lead)
- Carey Kireyev
- Parag Mhashilkar
- Nate Mueller
- UW-Milwaukee
- Scott Koranda (Alum)
- Brian Moe
- Univ. of Chicago
- Xin Zhao
- Caltech
- Erik Aslakson
- CMS
- Rick Cavanaugh
- Ruth Pordes
- Atlas
- Kaushik De
- Saul Youssef (Pacman)
- And a host of supporting people
10The UW VDT team
11Where do the tools come from?
NMI
VDT
Test
Sources (CVS)
Build
Binaries
Build Test Condor pool (37 computers)
Pacman cache
Package
Patching
RPMs
Build
Binaries
GPT src bundles
Build
Binaries
Test
Contributors (VDS, etc.)
12Testing
- Thorough testing is essential for high-quality
releases - We do automated nightly testing
- Installation procedure
- Functionality of software
- We work closely with National Middleware
Initiative (NMI) of the NSF to leverage their
testing procedures and facilities - All releases tested by iVDGL VDT Testers Group
before distribution
13Upcoming Releases
- VDT 1.1.14
- Testing release within a week, release in 2-3
weeks - Switch to Globus 2.4.3
- VDT 1.2
- Early Summer 2004
- Based on VDT 1.1.14
- 1.2.x series will be bug fixes only, meant for
production - VDT 1.3
- Early Summer 2004
- Based on web-services Globus (GT 3.2, 4.0)
- Will allow users to experiment with new Grid
standards
14The Grid3 experience
- Met Metrics in Nov 2003
Continuous use with mainly Application
Effort since Nov 6 months - Infrastructure is stable and Operations Center
support load is small. - US CMS has seen 50 increase in application
throughput through opportunistic use resources. 1
FTE of support from is factor of 2 less than in
2003. - US ATLAS use ramping up for DC2 without
perturbing ongoing use of Grid3. - Computer Science and Biology applications run
periodically and job mix seen to change without
impact on overall efficiency.
15LCG/EGEE link
- As of March 22, 2004, new liaison between VDT and
LCG/EGEE funded by NSF - Resolves bugs reported by LCG/EGEE
- Add capabilities to VDT needed by LCG/EGEE
- Integrate LCG/EGEE components into VDT
- Maintain UW component of LCG/EGEE certification
testbed - Full-time and co-managed
16What did we learn
- What it takes to manage and package a multi
vendor collection of sophisticated middleware - What it takes to deploy, support and upgrade such
a collection. - What it takes to test and trouble shoot such a
collection - What it takes to deliver end-to-end
capabilities in a distributes environment
Prototyping Cyber-infrastructure
17Challenges ahead
- How do we support transition between versions of
common grid middleware? (like GT2-GT3-GT4 ...) - How do we deal with evolution in the
infrastructure ( OS, HW, utilities, ) - How do we improve the readiness of the tools in
the VDT - How do we make it easier to deploy, configure and
maintain a complete cluster/grid site - How do we guarantee longevity
- How do we scale the support infrastructure
- How do we scale the middleware engineering effort
18Where do you learn more about VDT?
- http//www.griphyn.org/vdt
- Support
- Alain Roy roy_at_cs.wisc.edu
- Miron Livny miron_at_cs.wisc.edu
- Ticket system vdt-support_at_ivdgl.org
19Grid Software Installation
Typical Grid Software Installation Experience
VDT Installation Experience!