Grappa Progress

1
Grappa Progress
Shava Smallen Dan Engh LBNL Grid Workshop July
29-31, 2002
2
Grappa

• Based on Indiana XCAT Science Portal Technology
• Jakarta/Https (secure)/Java/Globus (cog)
• html/python scripting
• Provides Unifying Framework
• Based on Internet/Grid-communication protocols
• Configurable notebooks
• Athena Notebook provides
• Visual Access to Grid Resources
• Compute Resources
• MAGDA Catalogue
• Visual Framework for Job Submission
• Script-based Framework for Job Submission

3
Grappa

• Grappa Brief History
• Design Proposals Spring 2001
• First Atlas Demo March 2002
• Selected (May 2002) as framework for SC Demo

4
Grappa Development Team

• Primary Developers
• Shava smallen
• Lisa Ensman
• Rob Gardner
• Rob Kopenec
• Dan Engh
• Primary Testing help
• Ed May
• Jerry G.
• Saul Youssef
• Magda Coordination
• Wensheng Deng

5
Some Milestones

• Milestones
• Submission across entire testbed (Feb 2002)
• Large Scale Job submission 50Mevents (April
  2002)
• Integration with MAGDA (May 2002)
• Registration of metadata with MAGDA (June 2002)
• Simplified/Decentralized Setup (July 2002)
• Resource Weighted Scheduling (July 2002)
• Script based cronjob (July 2002)

6
Grappa / Athena

• Has been run with local Athena installation
• Has been run with AFS implementation
• Has been run with static boxed versions
• Atlfast
• Atlsim

7
Current Capabilities

• Interactive Functionality (Web pages)
• User Configuration and Submission of Jobs
• Web based Job monitoring of all jobs
• (web launched or script launched)
• Script-base functionality
• Interactive script submission
• Automatic script submission

8
Grappa Testbed

• Currently about 60 CPUs available
• 5 condor pools
• IU, OU, UTA, BNL, BU
• 2 standalone machines
• ANL, UMICH
• LBL not tested
• Grappa/testbed production rate, cron-based,
• currently running at 15Mevents/day.

9
Typical User session
Start up portal on selected machine Start up
web browser Configure/Select testbed
resources Configure input files. Submit job
10
User Session
Open monitoring window Auto refresh (User
configurable) Monitor/cancel jobs
11
User Session
Monitor Cluster health By the way the new
ganglia version creates an additional level
combining different clusters into a
metacluster Upgrade in progress at IU and BNL
to be combined into the USATLAS metacluster
12
User Session
Browse MAGDA Catalogue Search for
Personal files Search for Physics Collections
(based on metadata to be added soon)
13
Production Running

• Configure Cronjob
• Production Input files
• Writing to magda cache
• Automatic subnotebook naming
• Setup up cron script location and timing
• Script interacts with same portal as web-based
• Use interactive mode to monitor production

14
Production Monitoring
Configure and test scripts command line
submission view log files
15
Production Monitoring
Log into web-portal cron submissions appear as
new subfolders click on subfolder to check what
was submitted monitor job status
16
Production Monitoring
Select jobs/groups of jobs to monitor auto
refresh (user configurable) cancel job option
17
Production Monitoring
Browse MAGDA catalogue Auto registration of
files check metadata (currently available as
command line tool) Search for files soon
search for physics collections
18
Metadata in MAGDA
Metadata published along with data files MAGDA
registers metadata Metadata browsing available
as command line tool -check individual files
19
Grappa communications flow
Script-Based Submisson interactive or cron-job
http - JavaScript
Cactus framework
https - JavaScript
Web Browsing Machine (JavaScript) Netscape/Mozilla
/Int.Expl/PalmScape
Input files
GLOBUS Submission,Monitoring
http// browse catalogue
Grappa Portal Machine XCAT tomcat server
Data Storage - Data Disk - HPSS
. . .
MAGDA registers file/location registers file
metadata
Magda
GLOBUS
Resource A
Resource Z
(spider)
Data Copy
Compute Resources
20
July Testbed experience

• 2 Grappa developers Shava,Dan
• Lots of testing from ANL, BU
• Testbed production efforts split
• Group efforts still being directed to GRAT
• Currently we have 2 systems, GRAT/GRAPPA
• These are not complimentary activities
• (2 SC demos?)
• GRAPPA has not been testedby all users

21
On Deck...

• Real Job Scheduler
• Condor-g
• Automated resource configuration
• MDS-based
• Visualization
• USATLAS ganglia metacluster
• MAGDA physics collections
• On line GRAPPA processes
• Debugging

22
Summary

• Grappa uses new technologies to coordinate a wide
  variety of Athena/Grid activity
• Uses Internet/Grid protocols
• Command line
• Web interface
• Both script based
• User-configurable notebooks
• Potentially could be configured to do a wide
  variety of Athena activities.