The Virtual Data Toolkit VDT

1
The Virtual Data Toolkit (VDT)

• Miron Livny
• University of Wisconsin - Madison
• miron_at_cs.wisc.edu
• First EGEE Conference
• April 18-22, 2004

2
VDTs Mission

• GriPhyN a primary GriPhyN deliverable will be
  a suite of virtual data services and virtual data
  tools designed to support a wide range of
  applications. The development of this Virtual
  Data Toolkit (VDT) will enable the real-life
  experimentation needed to evaluate GriPhyN
  technologies. The VDT will also serve as a
  primary technology transfer mechanism to the four
  physics experiments and to the broader scientific
  community.
• iVDGL VDT Robustification and Troubleshooting
  team (VRT) .... will not develop new software but
  instead will work to enhance the releases of the
  GriPhyN VDT software, as follows
• (1) develop and maintain a VDT test suite
• (2) define and implement a unified and fully
  integrated error reporting framework across all
  VDT components
• (3) equip the VDT with dynamically configurable
  event logging capabilities
• (4) extend the VDT with new components required
  for specific application purposes
• (5) help users to maintain and trouble shoot the
  VDT software,
• (6) provide documentation, and
• (7) create procedures and tools for reporting,
  tracking, and fixing bugs.

3
What is the VDT?

• A collection of software
• Common Grid middleware
• Virtual data software
• Utilities
• An easy installation mechanism
• Goal Push a button, everything you need to be a
  consumer or provider of Grid resources just works
• Two methods
• Pacman installs and configures it all
• RPM installs some of the software, no
  configuration
• A support infrastructure
• Coordinate bug fixing
• Help desk
• Understand community needs and wishes

4
What is the VDT?

• A highly successful collaborative effort
• VDT Working Group
• VDS (Chimera/Pegasus) team
• Provides the V in VDT
• Condor Team
• Globus Alliance
• NMI Built and Test team
• EDG/LCG
• Middleware, testing, patches, feedback
• PPDG
• Hardening and testing
• Pacman
• Provides easy installation capability
• Currently Pacman 2, moving to Pacman 3 soon

5
VDT Timeline Highlights

• Fall 2001 VDT started by GriPhyN and iVDGL
• Supported US-CMS testbed in 2002
• March 2002 VDT support system inaugurated
• Early 2003 Adopted by European Data Grid LHC
  Computing Grid
• April 2003 VDT Testers group started
• Fall 2003 Supporting Grid3
• Fall 2003 Adopted by Particle Physics Data Grid
• Nov 2003 Nightly test infrastructure deployed

6
VDT Growth
VDT 1.1.8 First real use by LCG
VDT 1.1.11 Grid2003
VDT 1.0 Globus 2.0b Condor 6.3.1
VDT 1.1.7 Switch to Globus 2.2
VDT 1.1.3, 1.1.4 1.1.5 pre-SC 2002
7
Tools in the VDT 1.1.13

• Condor Group
• Condor/Condor-G
• DAGMan
• Fault Tolerant Shell
• ClassAds
• Globus Alliance
• Job submission (GRAM)
• Information service (MDS)
• Data transfer (GridFTP)
• Replica Location (RLS)
• EDG LCG
• Make Gridmap
• Certificate Revocation List Updater
• Glue Schema/Info prov.

• ISI UC
• Chimera Pegasus
• NCSA
• MyProxy
• GSI OpenSSH
• UberFTP
• LBL
• PyGlobus
• Netlogger
• Caltech
• MonaLisa
• VDT
• VDT System Profiler
• Configuration software
• Others
• KX509 (U. Mich.)

8
The PPDG factor

• PPDG played a key role in the hardening of VDT
  (US-CMS testbed)
• PPDG experiments (BaBar, STAR, Run II, US LHC,
  JLAB) have  adopted the Virtual Data Toolkit
  (VDT) packaging and distribution of the common
  Grid middleware.
• As part of the extension proposal (8/04-7/06),
  PPDG will -
• Align, to the extent possible, middleware
  deployments and timelines with the testing and
  release procedures of the VDT.
• Contribute to testing of VDT by integrating
  pre-release and development versions into their
  test beds for end-to-end validation and system
  tests.
• Adopted  services most likely to follow common
  Web-Service based interfaces. The project  will
  work with the VDT on service-based components  as
  they emerge and evolve.

9
Core VDT Working Group

• Miron Livny (lead)
• UW-Madison
• Alain Roy (technical lead)
• Carey Kireyev
• Parag Mhashilkar
• Nate Mueller
• UW-Milwaukee
• Scott Koranda (Alum)
• Brian Moe
• Univ. of Chicago
• Xin Zhao

• Caltech
• Erik Aslakson
• CMS
• Rick Cavanaugh
• Ruth Pordes
• Atlas
• Kaushik De
• Saul Youssef (Pacman)
• And a host of supporting people

10
The UW VDT team
11
Where do the tools come from?
NMI
VDT
Test
Sources (CVS)
Build
Binaries
Build Test Condor pool (37 computers)
Pacman cache
Package
Patching

RPMs
Build
Binaries
GPT src bundles
Build
Binaries
Test
Contributors (VDS, etc.)
12
Testing

• Thorough testing is essential for high-quality
  releases
• We do automated nightly testing
• Installation procedure
• Functionality of software
• We work closely with National Middleware
  Initiative (NMI) of the NSF to leverage their
  testing procedures and facilities
• All releases tested by iVDGL VDT Testers Group
  before distribution

13
Upcoming Releases

• VDT 1.1.14
• Testing release within a week, release in 2-3
  weeks
• Switch to Globus 2.4.3
• VDT 1.2
• Early Summer 2004
• Based on VDT 1.1.14
• 1.2.x series will be bug fixes only, meant for
  production
• VDT 1.3
• Early Summer 2004
• Based on web-services Globus (GT 3.2, 4.0)
• Will allow users to experiment with new Grid
  standards

14
The Grid3 experience

• Met Metrics in Nov 2003
  Continuous use with mainly Application
  Effort since Nov 6 months
• Infrastructure is stable and Operations Center
  support load is small.
• US CMS has seen 50 increase in application
  throughput through opportunistic use resources. 1
  FTE of support from is factor of 2 less than in
  2003.
• US ATLAS use ramping up for DC2 without
  perturbing ongoing use of Grid3.
• Computer Science and Biology applications run
  periodically and job mix seen to change without
  impact on overall efficiency.

15
LCG/EGEE link

• As of March 22, 2004, new liaison between VDT and
  LCG/EGEE funded by NSF
• Resolves bugs reported by LCG/EGEE
• Add capabilities to VDT needed by LCG/EGEE
• Integrate LCG/EGEE components into VDT
• Maintain UW component of LCG/EGEE certification
  testbed
• Full-time and co-managed

16
What did we learn

• What it takes to manage and package a multi
  vendor collection of sophisticated middleware
• What it takes to deploy, support and upgrade such
  a collection.
• What it takes to test and trouble shoot such a
  collection
• What it takes to deliver end-to-end
  capabilities in a distributes environment

Prototyping Cyber-infrastructure
17
Challenges ahead

• How do we support transition between versions of
  common grid middleware? (like GT2-GT3-GT4 ...)
• How do we deal with evolution in the
  infrastructure ( OS, HW, utilities, )
• How do we improve the readiness of the tools in
  the VDT
• How do we make it easier to deploy, configure and
  maintain a complete cluster/grid site
• How do we guarantee longevity
• How do we scale the support infrastructure
• How do we scale the middleware engineering effort

18
Where do you learn more about VDT?

• http//www.griphyn.org/vdt
• Support
• Alain Roy roy_at_cs.wisc.edu
• Miron Livny miron_at_cs.wisc.edu
• Ticket system vdt-support_at_ivdgl.org

19
Grid Software Installation
Typical Grid Software Installation Experience
VDT Installation Experience!