Title: The Open Grid Computing Environments Project
1The Open Grid Computing Environments Project
- Marlon Pierce
- Community Grids Laboratory
- Indiana University
2Acknowledgements
- Funding from NSF NMI (2003-2007) and OCI SDCI
(2007-2010). - Current participants
- Indiana University (Pierce, Gannon)
- RENCI (Kandaswamy)
- RIT(von Laszewski)
- SDSC (Wilkins-Diehr)
- SDSU (Thomas, Edwards)
- TACC (Dahan)
3Outline
- Web Portals and Science Gateways
- OGCE efforts
- OGCE Portal Software
- Portal tools
- Java COG, GTLAB
- OGCE Gateway Services
- GFAC, GPIR
- Software Engineering Issues
- What is next?
4OGCE Goals
- To provide easily installable, well-tested
software for building Web client and service
components that constitute a Grid Computing
Environment. - Science Web Portal --gt GCE --gt Science Gateway
- To support developing groups through training,
outreach, and divine intervention. - Gateways have many needs that cant be solved by
downloadable software alone.
5What Is a Web Portal?
- Aggregate content from multiple sources into a
single display. - Typically consume RSS/Atom news feeds.
- More powerful versions these days support Flickr,
calendars, games, etc. - Gadgets, widgets
- Examples iGoogle, Netvibes, My Yahoo!
6Science Portals and Gateways
- Science portals resemble standard portals, but
must also - Support access to computing and storage
resources. - Allow users remote, Unix-like access to these
resources. - Provide access to science applications and data
sets. - So security is crucial.
- And we must provide value added services as well
as user interfaces.
7A Comprehensive Gateway Architecture
8Components for Science Portals
- OGCE is founded on the principal that portals
should be built out of reusable parts. - Key standard in our first phase the JSR 168
portlet specification. - Portlets can run in multiple containers
- uPortal, Sakai, GridSphere, LifeRay, etc.
- Allows us to build Grid specific components and
deploy along side other goodies Sakai
collaboration tools, contributed portlets, etc.
9OGCE Portal Software
10OGCE GPIR portlet can interoperate with TeraGrid
and your own GPIR services.
11Manage TeraGrid MyProxy credentials with the OGCE
ProxyManager portlets.
12OGCE file management client portlets interact
with TeraGrid GridFTP servers.
13General purpose batch and interactive job
submission to GRAM, WS-GRAM is supported.
14Dashboard Portlet
The dashboard portlet allows users to track jobs
on the selected resource. The user can view
either his own set of jobs or get information on
all submitted jobs.
14
15(No Transcript)
16Queue forecasting portlets work with the NWS
QBETS to predict wait times and deadlines.
17PURSe portlets manage user requests for portal
accounts and Grid credentials.
18Condor and Condor-G
19OGCE IFrame Portlet can be used to integrate
external sites.
20Building Your Own Grid Portlets
21Coding Portlets
- Portlets are just servlet-like Java classes.
- Basic API key methods
- doView(), processAction().
- These are coupled to JSP pages (typically)
through tag libraries and request dispatchers. - OGCE supports Velocity portlets
- So we must provide the coding logic for
processAction(). - COG abstraction layers provide this.
22CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications
GT2
GT3 (X)
GT4 WS-RF
Condor
Unicore
SSH
Others
23Task
Task Handler
Task Specification
The class diagram is the same for all grid tasks
(running jobs, modifying files, moving data).
Service
Security Context
Service Contact
Classes also abstract toolkit provider
differences. You set these as parameters GT2,
GT4, etc.
24Task and Specification
- Task tasknew TaskImpl(mytask,
- Task.JOB_SUBMISSION)
- task.setProvider(GT2)
- JobSpecification spec
- new JobSpecificationImpl()
- spec.setExecutable(rm)
- spec.setBatchJob(true)
- spec.setArguments(-r)
-
- task.setSpecification(spec)
25Service and Security Context
- Service servicenew
- ServiceImpl(Service.JOB_SUBMISSION)
- service.setProvider(GT2)
- SecurityContext securityContext
- CoreFactory.newSecurityContext(GT2)
- //Use cred object from ProxyManager
- securityContext.setCredentials(cred)
- service.setSecurityContext(
- (SecurityContext)securityContext)
26Service Contact and Submit
- ServiceContact serviceContact
- new ServiceContact(myhost.myorg.org)
- service.setServiceContact(serviceContact)
- task.setService(
- Service.JOB_SUBMISSION_SERVICE,
- service)
- TaskHandler handlernew GenericTaskHandler()
- handler.submit(task)
27Coupling CoG Tasks
- The COG abstractions also simplify creating
coupled tasks. - Tasks can be assembled into task graphs with
dependencies. - Do Task B after successful Task A
- Graphs can be nested.
28Problems with Portlet Development
- Grid portlets typically wrap each single Grid
capability in a separate portlet - Problem is that Grid portlets need to combine
these operations - Portlets are entire web applications, so we need
a component model for portlets reusable portlet
parts - Even with the COG Abstraction Layer, we must
still do a lot of coding to biuld new
applications. - To address these problems we have adopted Java
Server Faces - Provides several nice Model-View-Controller
features - JSF provides an extensible framework (tag
libraries) for making reusable components. - Apache JSF portlet bridge allows you to convert
standalone JSF applications (development phase)
into portlets (deployment phase).
29Grid Tag Libraries and Beans (GTLAB)
- GTLAB provides common components for building
portlets using tags and reusable parts. - The goal of GTLAB to simplify Grid portlet
development - Enable rapid development
- GTLAB capabilities include Grid operations with
XML based tags within Java Server Faces (JSF)
framework. - Grid tag libraries are built using JSF custom
component development techniques - Grid tags are interfaces to backing Grid beans
- End users pass values to Grid beans by using tag
attributes. - We build on Java CoG 4s abstraction layer.
- Each backing Grid bean has equal capability with
a portlet application in case of Grid portlet
approach.
29
30GTLAB Example
- Grid tags are associated with Grid services via
Grid beans - Grid Beans wrap the Java COG Kit (version 4)
- We show an example JSF page section below.
- This allows you to develop new Grid portlets
with no additional Java code.
- lthtmlgt
- ltbodygt
- ltfformgt
- ltosubmit idtest actionnext_page /gt
- ltomyproxy idpr hostnamegf1.ucs.indiana.edu
port7512 lifetime2 usernamemnacar
password /gt - ltojobsubmit idtask hostnamecobalt.ncsa.tera
grid.org providerGT4 executable/bin/ls
stdouttmp/result stderrtmp/error /gt - lt/osubmitgt
- lt/fformgt
- lt/bodygt
- lt/htmlgt
30
31Grid Tags Associated Grid Beans Features
ltsubmit/gt ComponentBuilderBean Creating components, job handlers, submitting jobs
lthandler/gt MonitorBean Handling monitoring page actions
ltmultitask/gt MultitaskBean Constructing simple workflow
ltdependency/gt MultitaskBean Defining dependencies among sub jobs
ltmyproxy/gt MyproxyBean Retrieving myproxy credential
ltfileoperation/gt FileOprationBean Providing Gridftp operations
ltjobsubmission/gt JobSubmitBean Providing GRAM job submissions
ltfiletransfer/gt FileTransferBean Providing Gridftp file transfer
ResourceBean Describes common properties among all tags and beans. Passing values given by standard visual JSF components.
32How to prepare application pages
- Developers embed Grid tags snippet into JSF page
- These components are non-visual and are not
displayed in HTML. - Resource bean provides bridging with form inputs
and GTLAB framework. - lthoutputText value"Taskname "/gt   lthinputText
value"resource.taskname" /gt   ltomultitask
id"multi" persistent"true" task name"resource
.taskname" /gt - Dynamic values to Grid tag attributes are
provided by Resource bean. - Only visual component is ltosubmit/gt tag that is
associated with action method of GTLAB.
32
33GTLAB Dashboard PortletExample
- ltosubmit idtrack actionlist_page /gt
- ltomultitask iddashboard tasknametrack
persistenttrue gt - ltomyproxy idproxy hostnamegf1.ucs.indian
a.edu lifetime2 - usernameresource.usern
ame passwordresource.password /gt -
- ltojobsubmit idjobA
hostnamecobalt.ncsa.teragrid.org - providerGT4
executable/bin/whoami - stdouttmp/result
- stderrtmp/error /gt
- Â
- ltojobsubmit idjobB
hostnamecobalt.ncsa.teragrid.org - providerGT4
executable/bin/showq - stdintmp/result
stdouttmp/list - stderrtmp/error /gt
- Â
- ltodependency iddepend taskjobB
dependsOnjobA /gt - lt/omultitaskgt
- lt/osubmitgt
33
34Tracking and Managing Jobs
- GTLAB manages lifecycles of jobs and monitor
their status. - Grid operations are usually batch processes
- We provide callback mechanism to follow up the
jobs - GTLAB creates handlers for jobs and persistently
stores them. - GTLAB handlers manages the job events such as
stop, cancel or resuming the running jobs. - GTLAB provides archive for job metadata and
allows managing the archive - Handler tag helps to organize users job
repository - ltohandler iddelete action"monitor.delete"
gt - ltfparam id"task" name"taskname
value"task"/gt - lt/ohandlergt
34
35OGCE Gateway Services
36Web Services in Scientific Communities (G.
Kandaswamy)
- Web services are used to wrap scientific
applications to - Describe, publish, discover and consume
scientific applications in a standard way - Compose complex workflows from scientific
applications - Run and monitor complex workflows on distributed
resources - Such web services that wrap scientific
applications are called application services
36
37A Simple Application Service
Command-line Arguments
SOAP Request
Output Results
SOAP Response
Host1
Host2
37
38Things Are Usually More Complicated
38
39The Problem
- Application services may not be available during
a workflow execution - Unreliable resources (software, computers,
networks) - Heavy load on service
- Does not meet QoS or security requirements of
client - Workflows cannot complete unless all services are
available
39
40GFAC Solution
- A Generic Application Factory
- A persistent web service that knows how to create
instances of any application service - Use a Generic Application Factory to create
instances of application services on-demand from
workflows
40
41Implementation
- The Generic Application Factory (GFac)
- The Generic Service Toolkit A toolkit that
wraps any command-line application as an
application service - Without writing any web service code
- Without modifying the application in any
significant way
41
42Creating an Application Service (1/2)
- Write ServiceMap document to describe your
service - Write Application Deployment Description
document to describe a deployment of your
application - Upload the above two documents to a Registry
service
42
43Creating an Application Service (2/2)
Portal
Service Provider
Host1
1. Create service request
2. Get ServiceMap Host Description
3. Create service
4. Configure service
5. Register WSDL
5. Register capabilities
Host2
43
44Invoking an Application Service
Portal
5. Get Application Deployment Description and
Host Description
Host2
User
2. Access service
3. Return user interface
4. Invoke Service
7. Return results
4. Run application
6. Send notifications
Host3
44
45Software Engineering Issues
46OGCE Code Repository
- We use SourceForge, SVN
- http//sourceforge.net/projects/ogce
- Other SourceForge tools are useful.
- Replaced old OGCE bugzilla with SF bugzilla
recently after we were attacked by robots.
47Portal Build System
- The portal download gives you everything you need
to get started except Java. - Includes Tomcat, GridSphere, Ant, and Maven.
- Assume you have a Grid somewhere.
- Build system (recently revised) is designed to
build everything in one command. - mvn clean install
- Also designed to support extensibility (I.e.
replace GridSphere with Sakai) and simple updates
of portlets. - We use Maven 2 exclusively.
- Nice for managing third party jar dependencies.
- It can call Ant as necessary
- Testing portals is another matter
- Normal unit test systems like Junit are not
really appropriate.
48JMeter Test Suite
File Transfer portlet unit tested in JMeter UI
check for valid HTML response
Create lots of unit tests, run, and see results
in a dashboard
49Nightly Builds and Tests on NMI Testbed
50Whats Next?
51Some Future Issues
- Better support for science tools, not just bare
grids. - Experiment builder, Xbaya workflow manager,
metadata repository services and clients. - Better support for TeraGrid Science Gateways
- Logging, auditing, integration with GridShib
- JavaScript Grid abstraction layers and agent
services to support non-portlet clients. - More projects obviously we are interested in
working with the OSG
52What About Web 2.0?
- This is another talk entirely.
- http//grids.ucs.indiana.edu/ptliupages/presentati
ons/Web20Tutorial_CTS.ppt - http//grids.ucs.indiana.edu/ptliupages/publicatio
ns/Web20ChapterFinal.pdf - See also recent OGF 19 and 21 Workshops.
- Join us at SC07 for the GCE07 Science Gateway
Workshop - 20 peer-reviewed or invited talks, with focus on
Web 2.0.
53More Information
- OGCE Web Site
- www.collab-ogce.org
- Announcements Atom Feed
- http//collab-ogce.blogspot.com/atom.xml
- Contact me mpierce_at_cs.indiana.edu
54Some Example Portals
55LEAD Gateway Portal
NSF Large ITR and Teragrid Gateway - Adaptive
Response to Mesoscale weather events -
Supports Data exploration,Grid Workflow
56TeraGrid User Portal
57User Portal Sharable Portlets
- Account Management
- view projects and allocation usage
- view system account usernames
- view DNs registered for account
- add users to projects
- supports gt3500 users
- Resource
- view comprehensive list of TG resources and their
attributes - view job queues, load, status of resources
- Documentation
- current User Info documentation
- contextual help for all interfaces
- Consulting
- TG help desk information
- portal feedback channel
- Allocation
- Info about how to apply for/renew allocations
58 North Carolina Bioportal
- Principal collaborators John McGee and Lavanya
Ramakrishnan - Features
- access to common bioinformatics tools
- extensible toolkit and infrastructure
- OGCE and National Middleware Initiative (NMI)
- leverages emerging international standards
- remotely accessible or locally deployable
- packaged and distributed with documentation
- National reach and community
- TeraGrid deployment
- Portals hosted at RENCI and NCSA
- Education and training
59UNC-Charlotte Visual Grid Portal
Project Lead Prof. Barry Wilkinson Portal
Developer Jeremy Villalobos
60(No Transcript)
61(No Transcript)