The Virtual Data Toolkit VDT - PowerPoint PPT Presentation

1 / 19
About This Presentation
Title:

The Virtual Data Toolkit VDT

Description:

The development of this Virtual Data Toolkit (VDT) will enable the real-life ... 1.2.x series will be bug fixes only, meant for production. VDT 1.3. Early Summer 2004 ... – PowerPoint PPT presentation

Number of Views:76
Avg rating:3.0/5.0
Slides: 20
Provided by: Mar5351
Category:

less

Transcript and Presenter's Notes

Title: The Virtual Data Toolkit VDT


1
The Virtual Data Toolkit (VDT)
  • Miron Livny
  • University of Wisconsin - Madison
  • miron_at_cs.wisc.edu
  • First EGEE Conference
  • April 18-22, 2004

2
VDTs Mission
  • GriPhyN a primary GriPhyN deliverable will be
    a suite of virtual data services and virtual data
    tools designed to support a wide range of
    applications. The development of this Virtual
    Data Toolkit (VDT) will enable the real-life
    experimentation needed to evaluate GriPhyN
    technologies. The VDT will also serve as a
    primary technology transfer mechanism to the four
    physics experiments and to the broader scientific
    community.
  • iVDGL VDT Robustification and Troubleshooting
    team (VRT) .... will not develop new software but
    instead will work to enhance the releases of the
    GriPhyN VDT software, as follows
  • (1) develop and maintain a VDT test suite
  • (2) define and implement a unified and fully
    integrated error reporting framework across all
    VDT components
  • (3) equip the VDT with dynamically configurable
    event logging capabilities
  • (4) extend the VDT with new components required
    for specific application purposes
  • (5) help users to maintain and trouble shoot the
    VDT software,
  • (6) provide documentation, and
  • (7) create procedures and tools for reporting,
    tracking, and fixing bugs.

3
What is the VDT?
  • A collection of software
  • Common Grid middleware
  • Virtual data software
  • Utilities
  • An easy installation mechanism
  • Goal Push a button, everything you need to be a
    consumer or provider of Grid resources just works
  • Two methods
  • Pacman installs and configures it all
  • RPM installs some of the software, no
    configuration
  • A support infrastructure
  • Coordinate bug fixing
  • Help desk
  • Understand community needs and wishes

4
What is the VDT?
  • A highly successful collaborative effort
  • VDT Working Group
  • VDS (Chimera/Pegasus) team
  • Provides the V in VDT
  • Condor Team
  • Globus Alliance
  • NMI Built and Test team
  • EDG/LCG
  • Middleware, testing, patches, feedback
  • PPDG
  • Hardening and testing
  • Pacman
  • Provides easy installation capability
  • Currently Pacman 2, moving to Pacman 3 soon

5
VDT Timeline Highlights
  • Fall 2001 VDT started by GriPhyN and iVDGL
  • Supported US-CMS testbed in 2002
  • March 2002 VDT support system inaugurated
  • Early 2003 Adopted by European Data Grid LHC
    Computing Grid
  • April 2003 VDT Testers group started
  • Fall 2003 Supporting Grid3
  • Fall 2003 Adopted by Particle Physics Data Grid
  • Nov 2003 Nightly test infrastructure deployed

6
VDT Growth
VDT 1.1.8 First real use by LCG
VDT 1.1.11 Grid2003
VDT 1.0 Globus 2.0b Condor 6.3.1
VDT 1.1.7 Switch to Globus 2.2
VDT 1.1.3, 1.1.4 1.1.5 pre-SC 2002
7
Tools in the VDT 1.1.13
  • Condor Group
  • Condor/Condor-G
  • DAGMan
  • Fault Tolerant Shell
  • ClassAds
  • Globus Alliance
  • Job submission (GRAM)
  • Information service (MDS)
  • Data transfer (GridFTP)
  • Replica Location (RLS)
  • EDG LCG
  • Make Gridmap
  • Certificate Revocation List Updater
  • Glue Schema/Info prov.
  • ISI UC
  • Chimera Pegasus
  • NCSA
  • MyProxy
  • GSI OpenSSH
  • UberFTP
  • LBL
  • PyGlobus
  • Netlogger
  • Caltech
  • MonaLisa
  • VDT
  • VDT System Profiler
  • Configuration software
  • Others
  • KX509 (U. Mich.)

8
The PPDG factor
  • PPDG played a key role in the hardening of VDT
    (US-CMS testbed)
  • PPDG experiments (BaBar, STAR, Run II, US LHC,
    JLAB) have  adopted the Virtual Data Toolkit
    (VDT) packaging and distribution of the common
    Grid middleware.
  • As part of the extension proposal (8/04-7/06),
    PPDG will -
  • Align, to the extent possible, middleware
    deployments and timelines with the testing and
    release procedures of the VDT.
  • Contribute to testing of VDT by integrating
    pre-release and development versions into their
    test beds for end-to-end validation and system
    tests.
  • Adopted  services most likely to follow common
    Web-Service based interfaces. The project  will
    work with the VDT on service-based components  as
    they emerge and evolve.

9
Core VDT Working Group
  • Miron Livny (lead)
  • UW-Madison
  • Alain Roy (technical lead)
  • Carey Kireyev
  • Parag Mhashilkar
  • Nate Mueller
  • UW-Milwaukee
  • Scott Koranda (Alum)
  • Brian Moe
  • Univ. of Chicago
  • Xin Zhao
  • Caltech
  • Erik Aslakson
  • CMS
  • Rick Cavanaugh
  • Ruth Pordes
  • Atlas
  • Kaushik De
  • Saul Youssef (Pacman)
  • And a host of supporting people

10
The UW VDT team
11
Where do the tools come from?
NMI
VDT
Test
Sources (CVS)
Build
Binaries
Build Test Condor pool (37 computers)
Pacman cache
Package
Patching

RPMs
Build
Binaries
GPT src bundles
Build
Binaries
Test
Contributors (VDS, etc.)
12
Testing
  • Thorough testing is essential for high-quality
    releases
  • We do automated nightly testing
  • Installation procedure
  • Functionality of software
  • We work closely with National Middleware
    Initiative (NMI) of the NSF to leverage their
    testing procedures and facilities
  • All releases tested by iVDGL VDT Testers Group
    before distribution

13
Upcoming Releases
  • VDT 1.1.14
  • Testing release within a week, release in 2-3
    weeks
  • Switch to Globus 2.4.3
  • VDT 1.2
  • Early Summer 2004
  • Based on VDT 1.1.14
  • 1.2.x series will be bug fixes only, meant for
    production
  • VDT 1.3
  • Early Summer 2004
  • Based on web-services Globus (GT 3.2, 4.0)
  • Will allow users to experiment with new Grid
    standards

14
The Grid3 experience
  • Met Metrics in Nov 2003
    Continuous use with mainly Application
    Effort since Nov 6 months
  • Infrastructure is stable and Operations Center
    support load is small.
  • US CMS has seen 50 increase in application
    throughput through opportunistic use resources. 1
    FTE of support from is factor of 2 less than in
    2003.
  • US ATLAS use ramping up for DC2 without
    perturbing ongoing use of Grid3.
  • Computer Science and Biology applications run
    periodically and job mix seen to change without
    impact on overall efficiency.

15
LCG/EGEE link
  • As of March 22, 2004, new liaison between VDT and
    LCG/EGEE funded by NSF
  • Resolves bugs reported by LCG/EGEE
  • Add capabilities to VDT needed by LCG/EGEE
  • Integrate LCG/EGEE components into VDT
  • Maintain UW component of LCG/EGEE certification
    testbed
  • Full-time and co-managed

16
What did we learn
  • What it takes to manage and package a multi
    vendor collection of sophisticated middleware
  • What it takes to deploy, support and upgrade such
    a collection.
  • What it takes to test and trouble shoot such a
    collection
  • What it takes to deliver end-to-end
    capabilities in a distributes environment

Prototyping Cyber-infrastructure
17
Challenges ahead
  • How do we support transition between versions of
    common grid middleware? (like GT2-GT3-GT4 ...)
  • How do we deal with evolution in the
    infrastructure ( OS, HW, utilities, )
  • How do we improve the readiness of the tools in
    the VDT
  • How do we make it easier to deploy, configure and
    maintain a complete cluster/grid site
  • How do we guarantee longevity
  • How do we scale the support infrastructure
  • How do we scale the middleware engineering effort

18
Where do you learn more about VDT?
  • http//www.griphyn.org/vdt
  • Support
  • Alain Roy roy_at_cs.wisc.edu
  • Miron Livny miron_at_cs.wisc.edu
  • Ticket system vdt-support_at_ivdgl.org

19
Grid Software Installation
Typical Grid Software Installation Experience
VDT Installation Experience!
Write a Comment
User Comments (0)
About PowerShow.com