HotPage Portlet Development at TACC

1
HotPage Portlet Development at TACC

• Eric Roberts
• Texas Advanced Computing Center

2
Talk Outline

• Historical Overview
• Motivation
• Portlets GridPort 2.3
• Current portlets
• Difficulties with Jetspeed
• Future Directions

3
History

• Current HotPage portals
• PACI, NPACI portals
• Interface written in Perl / CGI
• Accessed via Web Browser
• Perl CGI has limitations but weve still managed
  to come quite far with it
• Telescience / BIRN portal
• Remote instrumentation
• GAMESS portals
• Chemical modeling

4
Why Move To Portlet Technology?

• More Complex, but todays grids require these
  capabilities
• The need for a more powerful portal technology
• Perl HotPage lacks
• Object-oriented framework
• Object persistence / caching
• Portlets offer these plus
• Customizable user interface
• Pluggable, reusable components (portlets)

5
HotPage 3.0 alpha

• Implemented using Jetspeed and portlet
  technologies
• Currently wrapping GridPort 2.3.1
• Use a thin CGI wrapper layer to communicate
  between Jetspeed and GridPort
• Uses Globus Toolkit 2.2 / GSI security
• Leverages GridPort Information Repository (GPIR)
  Query Web Services for informational data
• Still need to use Perl / CGI to interact with
  current GridPort 2.3 release

6
Why Jetspeed?

• Open Source
• Its free
• Jakarta project has large growing developer
  community
• Portability
• Share portlet code
• Growing Acceptance
• Used by other organizations in the Grid portal
  community
• Indiana University (Alliance portal)
• University of Michigan (CHEF Project)
• Being used in various other projects
• DoD PET portal
• SciDAC DOE Fusion portal
• OGCE portal

7
Architecture Portlets GridPort 2.3
Tomcat 4.1.x Web Server
GridPort 2.3 Services
Jetspeed
Globus 2.2
GridPort 2.3
Query WS
portlet
HTTP
HTTP
portlet
CGI Wrapper Layer
servlet
Client
servlets
8
Challenges Wrapping GridPort 2.3

• To track a users session in GridPort it sets
  cookies in the browser. However, it is
  complicated to track users with cookies using
  Jetspeed
• With help from Charles Severance from University
  of Michigan we figured out how to leverage the
  Tomcat session id to track users
• This involved minor modifications to GridPort in
  order to accept the session id if it was
  available
• This way GridPort 2.3 accommodates both old and
  new HotPage models

9
Jetspeed Login Servlet using GridPort 2.3
GridPort Certificate Repository
Client
GridPort creates proxy from certificate w/
username password
GridPort creates session file from session id
Tomcat Web Server running Jetspeed
Logged In!
Verify username / password with Jetspeed user
DB
GridPort creates session file from session id
Hypersonic SQL Database
10
Current Portlets

• Login/Logout Servlets
• GSI Authentication
• GridPort 2.3 wrapper portlets
• File Listing, File Upload
• Globusrun command execution
• Pass results inside XML and apply XSLT
• Also implemented using Velocity
• GPIR Informational Web Service portlets
• System Monitor
• Load, NodeMap, Jobs, MOTD, etc
• XML received from WS transformed with XSLT
• Implemented as Velocity portlets

11
GridPort 2.3 Wrapper Portlets
12
Manage Files

• Upload files to users portal space
• List files

13
Command Execution (using Globusrun)
14
GPIR Portlets

• System Monitor
• Static, Load Status and Job and Queue information
• Includes Job Information for individual resources
• Machine Summary Suite
• Customizable machine menu
• Add/remove machines from menu using Jetspeed
  parameter styles
• View NodeMap, Machine Information, Message of the
  Day, Load and Job/Queue Information for each
  machine

15
System Monitor
16
GPIR portlets
17
Currently Available

• Most recent build available at
• http//hotpage.tacc.utexas.edu2080/hotpage
• Capabilities
• Anonymous Informational View
• Grid View
• Individual Resources View
• File Management
• Command Execution

18
Roadmap

• Full implementation of GridPort 2.3 functionality
• Batch Job Submit
• Job Status
• Etc.
• Integrate with GridPort 3.0
• Pure Javano more Perl
• Administrative
• Account acquisition
• Other
• Integrate MyProxy, GridFTP and other portlets

19
Difficulties with Jetspeed

• Poor Documentation
• Lack of tools
• No IDE plug-ins for portlet testing like Sun ONE
  Studio or WebSphere
• Slow Development
• Open Source projects, although free, can have
  long development times
• Complex
• Steep learning curve for new portlet developers
• However, Jetspeed these challenges can be overcome

20
Future Directions

• OGSA Compliance
• What does it mean to be an OGSI client
• Integration with GridPort 3.0
• J2EE Integration
• More customization capabilities
• Struts and JSF

21
GridPort 3.0 Plans

• Tomislav Urban
• Texas Advanced Computing Center

22
Current GridPort 2.3

• Implemented in Perl
• Services are usual suspects
• Authentication/MyProxy
• Job Submission
• File Transfer
• Information Services
• Data Access (SRB)
• Approach
• Simple to install (Perl, little if any server
  side configuration)
• Light-weight
• Current Portals

23
NPACI HotPage
24
TACC HotPage
25
Telescience Access to Instruments/Data

• Uses GridPort to Integrate Telescience
  technologies with the Grid
• Access to instruments
• Globus job control
• SRB data collections
• Migrating to BIRN

26
Fusion Grid Portal

• GT2.x
• GP2.x
• Java CoG
• SRB
• JetSpeed Based Portal
• Will move to OGSA (GT3) when ready.

27
Current Status 2.x

• While moving on to GP3 design and development, we
  are continuing to develop new presentation
  technologies to work with GP2.x (e.g. portlets,
  web services)
• 2.x development and support will continue through
  2004
• NPACKage

28
Motivation for next Release
29
Motivation for next release

• Requirements Evolution
• Technology Evolution

30
Requirements Evolution

• Larger user community
• Campus Grid
• SciDAC/DOE (Fusion Grid)
• NPACI
• DOD
• State of Texas (TIGRE)
• NMI Testbed
• Collaborations
• Drives the need to support multiple VOs from a
  single GridPort instance with improved
  scalability and multi-portal management

31
Requirements Evolution

• Highly Distributed Users and Resources
• Drives the need to make GridPort services
  accessible over Web/Grid services
• Intelligent Workflow
• Web Services based workflow
• Need the ability to automate typical use
  scenarios, job submittal, file movement,
  visualization, etc.
• Remote Visualization
• Incremental Results
• Application Steering

32
Requirements Evolution

• Data
• Grid Meta Data
• The need for comprehensive gird meta-data
  including VO-Resource-Site relationships, not
  just Job and Load data
• Data Collection Access
• Integrate file and data collection access into
  GirdPort
• SRB integration supported under GP2.x will be
  kept for GP3.0

33
Requirements Evolution

• Generalized Grid Computing Environments (GCE)
• Application Profiling
• Need to enable the retention of run parameters
  across multiple Job runs
• Advanced Scheduling
• Integration with one or more Grid Meta-schedulers
• More Sophisticated User Interfaces
• Expanded Client Types
• Portals, PDAs, Shells (Command Line Interface)

34
Technology Evolution

• Many technological changes since original
  GridPort was conceived
• OGSI/OGSA
• GT3.0
• XML
• Java
• J2EE
• Jetspeed/Portlets

35
Grid Middleware Role

• Seems to be a lot of work ongoing in the space
  between low level grid software (e.g. Globus) and
  presentation (Portlets and Portals)
• This suggests that there may be distinction
  between high and low level grid services
• High-level middleware addresses
• The need to abstract low-level services (e.g.
  Globus)
• Integration of various Grid services under a
  uniform API (SRB, NWS, Globus, etc.)
• Service coordination
• Workflow
• User centric services
• Personalization (e.g. My Jobs, My
  Applications, etc.)
• Single sign-on
• Authorization

36
Approach
37
Grid Middleware Architecture

• Two possible philosophies
• Layer Approach
• Introduce a layer between low-level grid services
  and presentation
• Handles most or all business logic
• Coordinates inter-service flow
• Provides common services and centralized
  persistence
• May cause dependencies
• Collection of services
• Each service talks directly to low-level grid
  services
• May need to provide its own configuration and
  persistence mechanisms
• Each service remains independent and standalone

38
UI
UI

• Expanded support for a variety of UI clients
• Thin Client
• Browser
• Think Client
• Desktop Application (Client-Server)
• Command Line
• GCE Shell
• PDAs
• Functionality must be available beyond the
  portlet framework

Applications / Portals / Portlets
GridPort (Service Aggregation, Workflow)
OGSA Services
OGSI (GT3)
OS
Resources
39
Presentation

• Development of Portlet-based portals and
  applications
• This layer is reduced largely to presentation
  issues with GridPort or other high-level
  middleware driving most application logic
• Some Portlets may talk directly to low-level
  services

UI
Applications / Portals / Portlets
GridPort (Service Aggregation, Workflow)
OGSA Services
OGSI (GT3)
OS
Resources
40
GridPort

• Workflow interaction between OGSA Service
  components
• Component Approach
• need robust OO design ? Java
• RDBMS for persistence
• Goal is to present the Application/Portal
  developer with a simple unified API and built-in
  Application Logic (e.g. Workflow) to speed and
  aid rapid development

UI
Applications / Portals / Portlets
GridPort (Service Aggregation, Workflow)
OGSA Services
OGSI (GT3)
OS
Resources
41
Backend

• Distributed grid and web services (OGSA)
• Will support
• Security (GSI)
• Scheduling (LSF)
• Data Access (SRB)
• Job Submission (GRAM)
• Etc.
• Awaiting further definition of required grid
  services from GGF (OGSA-WG), this will drive the
  selection of which services to aggregate under
  GridPort

UI
Applications / Portals / Portlets
GridPort (Service Aggregation, Workflow)
OGSA Services
OGSI (GT3)
OS
Resources
42
J2EE

• GridPort 3.0 will be built on top of J2EE,
  leveraging the significant benefits of the J2EE
  platform
• Security
• Transactionality
• Distribution
• Persistence
• Scalability
• Failover
• Etc., etc.
• We are very interested in taking advantage of
  forthcoming JBoss and WebSphere OGSI integrations
• Write EJBs, expose Grid Services
• Portlets then interact with EBJs from within or
  without J2EE Web Container
• Would like to see JBoss Portlet implementation

43
GP3 Architecture
OGSA Services
Thin Clients
http
Security
OGSI
Data
Etc.
Thick Clients
RMI
GPIR
CMP
44
Grid Services

• GridPort 3 services will be exposed as OGSI
  compliant Grid Services for remote access
• They will also be available directly from within
  the J2EE web container (or via RMI) in order to
  ease scalability and performance concerns
• Use local direct calls where possible, Grid or
  Web Service calls only where necessary

45
GPIR

• All data supporting portal development will be
  cached in GPIR
• This data may come from a variety of sources, or
  be produced internally
• GPIR is simply GridPorts persistence mechanism,
  though it too is exposed as a web service
• GPIR focuses on multi VO Grid metadata
• Will be used to cache user sessions for
  statefullness across multiple portal visits

46
Shift in Approach

• The use of J2EE, DBMS, and Java represent a
  significant shift away from the light-weight
  approach of the Perl-based GridPort 2.x
• Significant server-side resources now required
• Significant increase in complexity
• This is mitigated by the availability of GridPort
  though a web/grid services interface
• Suggests two tiers of GridPort users
• User of an existing GP installation for a VO
• As currently, installation of ones own GridPort
  instance
• Simple and practical, remains the basic approach

47
Current Status 3.0

• Seeking to bring Software Engineering rigor to
  GP3 development
• Development Methodology Established
• UML
• Rational Unified Process
• Building Development Environment
• Dev, QA, Publish/Production
• Currently in Requirements Gathering phase

48
Summary

• Seek to have pre-alpha prototype complete by end
  of Fall 2003
• Will be using iterative development approach,
  starting with core (existing GP2.x) services
  implemented in the new architecture, then moving
  on to new features

49
Summary

• Bottom line
• GridPort seeks to aggregate core grid services in
  the interest of presenting simple, powerful and
  streamlined access to backend grid services and
  higher-order workflow to UI developers (Portals,
  Portlets, Shells, Applications, etc.)