Reusable Grid Portal Components

1
Reusable Grid Portal Components

• Mary Thomas

2
Outline

• The Portal Challenge
• The Solution
• The Open Grid Computing Environments Consortium
  (OGCE)
• The OGCE Portal Architecture
• Application Portal Examples
• NMI OGCE Plans

3
The Portal Challenge
4
Three-Tiered Architecture
Grid and Web Protocols
JDBC, Local, or Remote Connection
Portal User Interface
Database Service
Database
Portal Client Stub
Grid Resource Broker Service
HPC or Compute Cluster
Portal Client Stub
Information and Data Services
Grid Information Services, SRB
Portal Client Stub

• Three-tiered architecture is accepted standard
  for accessing Grid and other services

5
Portal are Gateways to Grid

• Early Grid Portals were great success
• Applications oriented,
• Provided scientists with some useful tools
• Encapsulated some complexities.
• But they were
• All stovepipe solutions.
• Very little reusable code and infrastructure
• Nothing could be shared between portals
• Many ignored emerging architecture of the Grid
• A framework based on distributed ubiquitous
  services

6
Impact of Distributed Grid and Web Services
Launch, configure And control
Open Grid Service Infrastructure (web service
component model)
Online instruments
(slide by Dennis Gannon)
7
Advantage of Portlet Architecture

• Portlet/component approach very compatible with
  Web/Grid services model
• Each Grid service can be associated with a unique
  portlet
• Very easy to add new services
• Many different Groups can contribute portlets
  which can be plugged into a portal.
• Currently Indiana, Argonne, Michigan, NCSA,Texas
  GridSphere portal group following this model.
  Most likely others as well.
• Jetspeed is Apache standard. Basis for IBM
  WebSphere portal. Also Oracle, Sun and others.
• Now a Java Standard (waiting for JSR 168)
• User can select and configure portlets he/she
  wishes to use
• Selection becomes part of persistent context

8
The Next Generation Grid Portal

• Building on Standard Technologies
• Portlet Design (JSR-168) IBM, Oracle, Sun, BEA,
  Apache
• Grid standards Java CoG, Web/Grid Services
• Web server JetSpeed (open source)
• User configurable, Service Oriented
• Philosophy The Portal is a gateway to
  distributed Grid and Web Services
• With common API, portlets can be exchanged,
  interoperate

9
User Requirements Have Not Changed

• Users Clients are
• Portal clients/users
• Portal developers
• Security
• Single sign-on, authentication to all grid
  resources
• Certificate based / uses MyProxy
• VO management
• File Management
• Grid FTP
• File staging
• Metadata management
• Remote Job Management
• Job submission, scheduling, and tracking. Log
  and output mgmt.
• Workflow management tools

10
User Requirements

• Information Services
• Grid resource and status information directories
• Data and metadata catalogs
• Personal and VO metadata and service
  index/directories
• Database Interfaces
• Application Interfaces
• Configure and launch workflow from domain
  specific interface
• Access to Collaboration Tools
• VO collaboration. Synchronous and asynchronous
• Access Grid venues
• Developers need common API

11
The SolutionThe Open Grid Computing Environment
(OGCE) Consortium
12
Open Grid Computing Environments (OGCE) Consortium

• Motivation
• Emerging technologies impacting portal
  developers
• Grid/Web services ubiquitous database use, XMl
• Lessons learned from GCE-RG and other projects
  (GridPort/HotPage, Alliance Portal, DOE
  Components, NEESGrid, Gateway, GRaDS, others)
• Multiple projects repeating similar tasks, work,
  development
• OGCE established in Fall 2003 to foster
  collaborations and sharable components with
  portal developers worldwide.

13
OGCE

• Funded by
• NSF NMI software releases will be part of NMI
  and synchronized with GRIDS releases
• Also NCSA Alliance, NPACI, DOE, NASA, others
• Release Schedule
• First Release 11-15-03
• Next Release April, 2004
• Website http//www.ogce.org

14
OGCE Tasks/Goals

• Development of interoperable Grid portal
  architecture
• Define standards for common APIs between portlet
  service and remote services
• Establishment of a Grid Portal Collaboratory
• Repository of portlet and portal service
  components
• An online forum for developers of Grid Portals
• The building of reusable portal components
• Can be integrated in a common portal container
  system
• Contribute to standards organizations
• NSF NMI program ? software releases
• GGF if appropriate or applicable

15
The OGCE Portal Architecture
16
OGCE Initial Architecture
Grid
Java
Protocols
Java
Local
CoG
COG
Grid Services
Portlets
API
Kit
GRAM,
MDS
-
LDAD
MyProxy
Grid Services
CoG
Proxy
Interfaces
Remote
Portlets
HTTP
Stubs
Other Services
Portal
SOAP
CHEF
Service
Teamlets
API
Services
Jetspeed
Internal
Services
Initial architecture aggregates multiple services
into a single portal using portlet containers

17
OGCE Initial Arch Features Capabilities

• OGCE approach encompasses multiple portal
  development systems
• Preserving development investments
• Central portal system (Jetspeed) acts as a
  central portal container
• Portal responsible for managing the lifecycle of
  pluggable portal components, or portlets.
• Grid/computational portlets must provide services
  on the behalf of the user
• Not just a collection of links/static pages
• User view (from the left) is of a single portal
  composed of portlets.

18
Supports Aggregation of Diverse Tasks

• Portlets provide a uniform management interface
  to the portal (on the left) connect to
  heterogeneous set of APIs on the right.
• First horizontal thread
• Portlets interact with web interfaces written in
  other languages such as Perl, Python, PHP,
  ColdFusion, etc.
• Architecture for bridged support is depicted in
  the second horizontal thread
• Examples
• Locally running Myproxy Grid portlet uses Java
  CoG kit to interact with remote Grid services
• Job Scheduler portlet uses GridPort Toolkit to
  submit job via Community Scheduling Frameworks
  (CSF) service
• Teamlets (sharable portlets) share collaboration
  and Grid services, such as calendars, message
  boards, file services.

19
OGCE - Revised Architecture
Grid Protocols
Portlets and Teamlets
Service API
Grid Service Stubs
Grid Services
Java CoG Kit
Local Portal Services
Remote Content Servers
Remote Content Services
HTTP
Jetspeed Internal Services

• Revised portal architecture provides a unified
  interface for portal services.
• Supports aggregation of services in backend

20
Features

• All services will employ a single (generic)
  service API that allows them to be deployed and
  accessed in a uniform fashion.
• Developer then chooses (during portal
  configuration) the implementations of particular
  services.
• Services may be local, may access remote sites
  via Grid protocols, or may access remote content,
  such as web interfaces written in Perl.
• OGCE Twist
• Entire portal will auto-install and include all
  required software components

21
Key Technologies????

• Core Portal Framework
• CHEF/JetSpeed (U. Mich)
• Core Grid Technologies
• JavaCoG (provides Globus interface)
• GridPort Toolkit (aggregates JavaCoG, and other
  grid services)

22
Portal/Application Gallery
23
TeraGrid Portal

• Access Point for TeraGrid Services
• Version 0 Collecting Initial Services
• Public Information about Resources
• Private Information for the developers.
• Version 1 User centered portal
• HotPage/GridPort style access to user accounts,
  credentials, job submission management.
• Version 2 Portals for Science Collaborations (Q3
  2004)
• Shared spaces, whiteboards, AG access, group
  authorization, shared application services

24
TeraGrid Integrated Architecture
Grid Services
Portlets and Teamlets
Grid Service Stubs
Service API
Java CoG Kit
GridPort Toolkit
Web Services
Local Portal Services
Remote Content Servers
Remote Content Services
HTTP
Jetspeed Internal Services

Diagram demonstrates how existing software
projects (such as GridPort) can be adapted to
support NMI Portals software system
25
Alliance scientific portal
Generate and store derived products
Execute application
Gather data files, resources
Create job description
Store and record results
Retrieve customized run description
template from data management services
Broker Service
Monitoring results captured in event services
persistent channel. myMEAD records address of
service and channel
Need to store y and z
Need x put to l by time y00
monitoring
Teragrid data service
Teragrid Data Service-single point of contact to
server-side data servers
Provides context to resolve x, y, z
myMEAD personal information space
Delayed invocation of GridFTP
26
DOE Fusion Grid Portal

• Portal Development on the DOE SciDAC Grid
• Initially, Fusion Grid
• Multi-site collaborative project
• TACC, Indiana, SDSC, General Atomics
• Developing interoperable portal and Web services
• Portal architecture based on NMI Portal software

27
LEAD Portal
28
CHEF - NEESGrid

• NEES Tools
• Data Viewers (Live/Stored)
• Telepresense
• Video as Data
• Electronic Notebook
• Data/Metadata Repository
• Grid Technologies
• Grid sign on using myproxy
• Grid computational portal
• GridFTP
• ..Many more

29
OGCE Plans
30
OGCE Portal System
OGSA Globus COG
NMI Dist
System Integration
CHEF Toolkit
Gridport
Alliance Portal
Testing
Workflow
Slide courtesy of Chuck Severance
31
NMI Features List (April 04 Release)

• Content Management
• Resources Users and groups can upload and share
  documents and URLs.
• News Several news feeds are available, including
  XMLHack and BBC News.
• Grid Context Store arbitrary web objects
  (movies, web pages, audio files) in a
  customizable, annotated directory tree.
• Collaborative
• Chat Engage other members of your group in
  online discussion.
• Discussion Participate in discussions with other
  members of your group
• Calendar Programschedule appointments and attach
  associated documents and URLs.  Calendars can be
  used to schedule both user and group activities. 
  
• Anabas Impromptu system for real time shared
  display, audio, and chats.

32
NMI Features List (April 04 Release)

• GT 3.0/Interactive
• Proxy Manager and Job Submit Obtain and manage
  one or more Grid credentials that can be used to
  access Grid resources through your browser.
• GridFTP Use Grid credential to browse remote
  directories and to upload and download files
  between your desktop and a remote host.
• Application Manager
• Application Factory Manager
• Remote Job Management
• GridAnt for workflow
• Job submission portal
• Community Scheduling Framework (CSF) submit jobs
  to scheduler
• Information Services
• LDAP Browser
• Navigate the LDAP server of your choice.
• GridPort Information Repository Portlets (GPIR)
  Portlets are available for browsing Machine
  Summaries,  System Status, and Total Grid
  Capabilities.
• Application Portlets
• Specialized portlets for projects

33
Key Feature Download a Portal Install

• Prerequisite Software
• Java 2 Standard Edition version 1.4.x or higher. 
• Apache Ant version 1.5.x or higher. 
• Preparing the Installation
• Optionallly, create a "portal" account on your
  web server.
• Download the portal bundle
• Put the download bundle into the home directory
  of "portal" account.
• Untar or unzip the portal bundle. 
• Globus installation process will create the
  directories .globus and .globus/certificates in
  the
• Software automatically installed includes
• Globus/Java CoG
• CHEF

• Install the Portal
• Move into the portal/chef-1.0.7 directory.
• Edit file "build.properites"
• Type "ant deploy" to build in and install the
  portal.
• Start the Tomcat web server. 
• Point your browser at http//ltyour host
  namegt10081/ogce and create an account.
• OS/Machines Tested
• MAC OSX
• PC/Windows XP
• Linux
• Cygwin

34
Current Status

• SC03 Demonstration portal and software download
  available
• Beta1
• Updated Release available now
• Bug fixes
• Now supporting GT3.x with J2SDK 1.4.2
• Working on NMI April 2004 release
• Including new features to support science portal
  applications
• Open GCE Runtime Engine (OGRE) workflow services
  and portlets
• Sample template portlets for simple science
  applications
• Based on aggregation of existing projects
• OGSA/OGSI/GT3.x compatible

35
Future Directions

• Moving towards new portlet standard
• JSR 168
• Should allow compatibility with
• GridSphere
• Academic education and business portals Oncourse,
  OneStart, CHEF (of course), and uPortal
• Major commercial products offerings from Sun,
  IBM, Oracle, etc.
• Incorporate new features
• Grid Services (CSF, OGSA-DAI portlets, Job
  Sequencer)
• Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions
• DOE SciDAC projects (such as CMCS)
• Expand online repository
• Develop process for third party contributions

36
Future Directions

• Moving towards new portlet standard
• JSR 168
• Should allow compatibility with
• GridSphere
• Academic education and business portals Oncourse,
  OneStart, CHEF (of course), and uPortal
• Major commercial products offerings from Sun,
  IBM, Oracle, etc.
• Incorporate new features
• Grid Services (CSF, OGSA-DAI portlets, Job
  Sequencer)
• Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions
• DOE SciDAC projects (such as CMCS)
• Expand online repository
• Develop process for third party contributions

37
Participants
38
References

• OGCE Website
• http//www.ogce.org
• NSF Middleware Initiative
• http//www.nsf-middleware.org