National Collaboratories Middleware Projects

1
National CollaboratoriesMiddleware Projects

• Ray Bair
• NC Program Coordinator
• January 15, 2002

2
Topics

• What is middleware?
• Goals of NC Middleware research
• Whats coming from the NC program
• Interacting with NC projects
• Tools you can use now
• Appendix More about middleware projects

3
Middleware software that connects or mediates
between two otherwise separate programs

• NC Middleware Program is focused on
• Technology to enable ubiquitous access to remote
  resources computation, information and
  expertise
• Capabilities that make it easier for distributed
  teams to work together, over the short and long
  term
• Standard services and protocols for access to
  networked resources, that aid software
  development/interoperability
• Middleware advances that enable scientific
  computing, e.g., high performance for scientific
  applications

4
Middleware Roles
5
DOE NC Middleware Projects

• Realizing the Science Grid

• Tools for Distributed Applications

6
Realizing and Enhancing the Science Grid

• Consistent access to resources and information
• High performance data transport
• Reliable and secure communication among peers
• Tools to manage data
• Tools to utilize metadata
• Metadata information about data or other
  information

7
Distributed Security Architectures Mary
Thompson, LBNL

• Concept
• Secure/flexible way to authorize access to
  distributed resources
• Based on signed Policy, Use-condition and
  Attribute certificates
• Stakeholders flexibly set access policy on a per
  resource granularity
• Existing Technologies
• Akenti 1.0beta, Secure Akenti-enhanced web server
  beta
• Sample secure ORBIX CORBA ORB with Akenti
  integration
• Future Capabilities
• Enhanced, secure Akenti-enabled Apache web server
• Web interface to generate UseCondition/Attribute
  certificates
• Proxy credential delegation capabilities
• Integration with GSI Grid Security
  Infrastructure

8
SciDAC DataGrid MiddlewareI. Foster, ANL C.
Kesselman, ISI Miron Livny, UW

• Concept
• Innovative techniques for co-reservation of Grid
  compute, network, and storage resources, and
  market brokering services
• High performance I/O, with intelligent, adaptive
  recovery
• Efficient, distributed replica management to
  improve access efficiency
• Existing Technologies
• Globus Toolkit, including GridFTP, GRAM,
  Condor
• Future Capabilities
• New Data Mover Technology supporting very high
  speed, reliable data movement, via new protocols
• Data Transfer Management for orchestrating data
  transfers
• Collective Data Management to support
  replication, mirroring, wide area hierarchical
  storage management

9
Reliable and Secure Group CommunicationDeb
Agarwal, LBNL

• Concept
• Many-to-many group communication that scales to
  the Internet
• Flexible message delivery in terms of reliability
  and ordering
• Peer-to-peer, secure, reliable, ordered multicast
  
• Existing Technologies
• InterGroup Protocol Design, and prototype
  implementation
• Future Capabilities
• Group Security Layer, for secure group
  communications
• Creates an SSL equivalent for group communication
• Software tools that support these protocols

10
Storage Resource Management for Data Grid
AppsArie Shoshani, LBNL Don Petravick, FNAL

• Concept
• Grid Middleware to support dynamic storage
  management for long lasting simulation and
  analysis tasks in a distributed environment
• Coordinate distributed disk caches by pinning
  of files
• Existing Technology
• Existing HPSS-HRM (Hierarchical Resource Manager)
  used by several HEP applications, based on
  logical file requests
• Future Capabilities
• Disk Resource Manager manages a single shared
  disk cache
• Storage Resource Manager manages what data is
  on each storage device
• Sophisticated pinning capabilities manages
  requests to cache files

11
Scientific Annotation MiddlewareJim Myers, Elena
Mendoza, PNNL Al Geist, Jens Schwidder, ORNL

• Concept
• Unified, lightweight metadata infrastructure to
  support the creation and use of metadata and
  annotations
• Annotations shared among portals and problem
  solving environments, software agents, scientific
  applications, electronic notebooks,
• Existing Technology
• DOE 2000 Electronic Notebooks
• Future Capabilities
• Search Tool, Metadata Viewer, Graphical
  Relationship Browser, Data Viewer, Data Signature
  Widget
• Notebook Explorer Viewer

12
Building Distributed Applications
Collaboratories, Portals, and Problem-Solving
Environments

• Use standard application components in portals
• Use Grid services from high level frameworks
• Tools for constructing scientific workflow
• Create virtual venues for collaborative work
• Portal a science-oriented PSE, typically with a
  web browser interface, that allows scientists to
  compose and run distributed applications, or to
  access and analyze distributed data.

13
Middleware Technology to Support Science
PortalsDennis Gannon, Randall Bramley, Indiana U.

• Concept
• A Science Portal that makes it easy to build and
  access Grid-based scientific applications from
  desktop web tools
• Organized as a set of Active Notebooks with web
  forms to launch and control the application, as
  well as histories
• Existing Technology
• GCE-WG Portal Software Repository, Indiana Active
  Notebook
• Future Capabilities
• Active Documents for science-centric input and
  analysis
• Active Notebooks contain grid-scripts and run
  histories, that can be modified and shared
• Component composition of application services

14
SciDAC Commodity Grid (CoG) KitsGregor von
Laszewski, ANL Keith Jackson, LBNL

• Concept
• Reusable Web Services that access underlying
  Grid services.
• Support rapid development of Science Portals,
  Problem Solving Environments, and applications
  that access Grid resources
• Existing Technology
• CoG Kit prototype
• Future Capabilities
• CoG access to basic Grid services
• GRAM, MDS, Security, co-scheduling,
• Support to portal development teams
• Components that are composable with a visual tool

15
Pervasive Collaborative Computing EnvironmentDeb
Agarwal, LBNL Miron Livny, U Wisconsin

• Concept
• Collaboration tools that support the continuum of
  collaborative interaction in a persistent
  environment
• Workflow tools that enable coordination of Grid
  computing processes and human tasks
• Existing Technology
• Range of DOE 2000 tools, university and
  commercial software
• Collaborative Virtual Workspace building/room
  metaphor
• Future Capabilities
• Building blocks for typical problems in
  scientific collaborations
• Workflow Framework
• Integrate messaging, Grid security, async.
  collaboration, Condor-G

16
Middleware to Support Group to Group
CollaborationRick Stevens, ANL

• Concept
• Collaborative work sharing beyond simple
  application sharing
• High end visualization envts integrated into
  collaborative spaces
• Extending asynchronous collaboration capabilities
  to embrace all types of data streams exchanged in
  a collaboration
• Existing Technology
• Access Grid
• Future Capabilities
• Scalable virtual venue server Improved AG
  security model
• Workspace Docking (app sharing) Easier node
  management
• Tiled display interfaces Multimedia
  record/playback

17
How do we get some of these things?Can we
influence the direction of projects?

• Interaction and feedback are fundamental elements
  of the NC program.
• All projects have dissemination plans.
• Learn project details at tomorrows Poster
  Session.
• Keep the dialogue going.

http//DOEcollaboratory.pnl.gov
18
NC Project Interactions
19
Supplemental Information

• More about individual middleware projects

20
Middleware Project Contacts and Web Sites

• Distributed Security Architectures
• Mary Thompson, MRThompson_at_lbl.gov
• High-Performance Data Grid Toolkit
• Ian Foster, foster_at_mcs.anl.gov
• Reliable and Secure Group Communication
• Deb Agarwal, DAAgarwal_at_lbl.gov,
  www-itg.lbl.gov/CIF/GroupComm/
• Storage Resource Management for Data Grid Apps
• Arie Soshani, arie_at_lbl.gov, sdm.lbl.gov/srm
• Scientific Annotation Middleware
• Jim Myers, jim.myers_at_pnl.gov, www.emsl.pnl.gov208
  0/docs/collab/sam/
• Middleware Technology to Support Science Portals
• Dennis Gannon, gannon_at_cs.indiana.edu
• CoG Kits
• Gregor von Laszewski, laszewski_at_mcs.anl.gov,
  www.cogkits.org
• Pervasive Collaborative Computing Environment
• Deb Agarwal, DAAgarwal_at_lbl.gov,
  www-itg.lbl.gov/Collaboratories/pcce.html
• Middleware to Support Group-to-Group
  Collaboration
• Rick Stevens, stevens_at_mcs.anl.gov,
  www-fp.mcs.anl.gov/fl/g2g/

21
Distributed Security Architectures
MICS Program Manager Mary Anne Scott
Principal Investigators Mary Thompson,
LBNL Funding Level 3 FTEs
Oct 23, 2001
22
SciDAC DataGrid Middleware
National Collaboratories Program

MICS Program Manager Mary Ann Scott
Principal Investigators I. Foster, ANL C.
Kesselman, ISI Miron Livny, UW
September 14, 2001
23
Reliable and Secure Group Communication
PI Orgs Logo
MICS/SciDAC Program Name
Picture/Diagram Related to Project
Principal Investigators Deb Agarwal - LBNL
MICS Program Manager Mary Anne Scott
Date Prepared
24
Storage Resource Managementfor Data Grid
Applications
Collaboratory Middleware

Principal Investigator Arie Shoshani, LBNL
Co-principal Investigator Don Petravick, FNAL
MICS Program Manager Mary Ann Scott
9/7/2001
25
Scientific Annotation Middleware
National Collaboratories Program
Novel Ideas
Proposed Timetable
Specification Alpha Release 1.0 Release 1.5 Release
Metadata Services 9/01 3/02 9/02 9/03
Semantic Services 12/01 7/02 12/02 7/04
Notebook Services 3/02 12/02 7/03 7/05
Interface Components 3/02 12/02 7/03 7/05
Pedigree Schema 9/02 7/03
Notebook Interface 3/02 12/03 7/04
Principal Investigators Jim Myers, Elena Mendoza
PNNL Al Geist, Jens Schwidder
ORNL
MICS Program Manager Mary Anne Scott
11/02/01
26
Middleware Technology to Support Science Portals
a Gateway to the Grid
MICS/SciDAC National Collaboratories and High
Performance Networks Middleware
Principal Investigators Dennis Gannon, Randal
Bramley, Department of Computer Science, Indiana
University
MICS Program Manager Mary Ann Scott
Date Prepared 1/02
27
SciDAC CoG Kits
MICS/SciDAC Program Name
Principal Investigators Gregor von Laszewski,
ANL
Keith. Jackson, LBL
MICS Program Manager Marry Ann Scott
09/07/2001
28
Pervasive Collaborative Computing Environment
PI Orgs Logo
MICS/SciDAC Program Name
Principal Investigators Deb Agarwal LBNL,
Miron Livny - UW
MICS Program Manager Mary Anne Scott
Date Prepared
29
Middleware to Support Group to Group Collaboration
MICS/SciDAC Middleware
Picture/Diagram Related to Project
Principal Investigators Rick Stevens, Argonne
National Laboratory
MICS Program Manager Mary Anne Scott
9/13/2001
30
Out Takes
31
Distributed Security Architectures Mary
Thompson, LBNL

• Novel Ideas
• Secure and flexible way to authorize access to
  distributed resources
• Based on signed Policy, Use-condition and
  Attribute certificates.
• Multiple stakeholders remotely control access to
  resources.
• Authorization server easily called from a
  resource gatekeepers
• Impact
• An authorization service, based on X.509 identity
  certificates and compatible with GSI/SSL
  connections, that can be easily used by
  distributed applications.
• Resource stakeholders can flexibly set access
  policy on a per resource granularity.

32
Distributed Security Architectures Mary
Thompson, LBNL

• Novel Ideas
• Secure and flexible way to authorize access to
  distributed resources
• Based on signed Policy, Use-condition and
  Attribute certificates.
• Multiple stakeholders remotely control access to
  resources.
• Authorization server easily called from a
  resource gatekeepers
• Impact
• An authorization service, based on X.509 identity
  certificates and compatible with GSI/SSL
  connections, that can be easily used by
  distributed applications.
• Resource stakeholders can flexibly set access
  policy on a per resource granularity.

33
SciDAC DataGrid MiddlewareI. Foster, ANL C.
Kesselman, ISI Miron Livny, UW

• Novel Ideas
• New protocols take advantage of unique properties
  of data grids
• Innovative techniques for co-reservation of
  compute, network, and storage resources, and
  market brokering services
• Variants of two-phase I/O strategies
• Intelligent, adaptive recovery and performance
  strategies
• Impact
• Common protocols services will insure
  interoperability of data grids
• APIs and SDKs implementing these protocols will
  be provided to allow easier access to data grid
  technology
• Efficient, distributed replica management will
  improve data access efficiency

34
Reliable and Secure Group CommunicationDeb
Agarwal, LBNL

• Novel Ideas
• Infrastructure to support true peer-to-peer
  communication
• Secure peer-to-peer group communication
• Reliable multicast capabilities that are scalable
  to the Internet
• Flexible message delivery in terms of reliability
  and ordering
• Impact
• Flexible communication infrastructure for
  collaborative applications that are truly
  peer-to-peer
• Many-to-many group communication that scales to
  the Internet
• Group layer that creates an SSL equivalent for
  group communication

35
Storage Resource Management for Data Grid
AppsArie Shoshani, LBNL Don Petravick, FNAL

• Novel Ideas
• Grid Middleware to manage storage resources
• Coordinate distributed disk caches by pinning
  of files
• Manage seamless access to tape storage
• Impact
• Provides essential component of Grid Middleware
• Will enable dynamic coordination of compute and
  storage resources
• Support storage management for long lasting
  simulation and analysis tasks in a distributed
  environment
• Manage job recovery from storage system and
  network failures, facilitating uninterrupted
  operation

36
Storage Resource Management for Data Grid
AppsArie Shoshani, LBNL Don Petravick, FNAL

• Novel Ideas
• Grid Middleware to manage storage resources
• Coordinate distributed disk caches by pinning
  of files
• Manage seamless access to tape storage
• Impact
• Provides essential component of Grid Middleware
• Will enable dynamic coordination of compute and
  storage resources
• Support storage management for long lasting
  simulation and analysis tasks in a distributed
  environment
• Manage job recovery from storage system and
  network failures, facilitating uninterrupted
  operation

37
Scientific Annotation MiddlewareJim Myers, Elena
Mendoza, PNNL Al Geist, Jens Schwidder, ORNL

• Novel Ideas
• Lightweight, flexible middleware to support the
  creation and use of metadata and annotations
• Sharing of annotations among portals and problem
  solving environments, software agents, scientific
  applications, and electronic notebooks
• Single unified metadata infrastructure
• Impact
• Improved completeness, accuracy, and availability
  of the scientific record
• Significant reduction of integration barriers
• An advanced notebook view of annotation data

38
Scientific Annotation MiddlewareJim Myers, Elena
Mendoza, PNNL Al Geist, Jens Schwidder, ORNL

• Novel Ideas
• Lightweight, flexible middleware to support the
  creation and use of metadata and annotations
• Sharing of annotations among portals and problem
  solving environments, software agents, scientific
  applications, and electronic notebooks
• Single unified metadata infrastructure
• Impact
• Improved completeness, accuracy, and availability
  of the scientific record
• Significant reduction of integration barriers
• An advanced notebook view of annotation data

39
Building Distributed Applications
Collaboratories, Portals, and Problem-Solving
Environments

• Use standard application components in portals
• Use Grid services from high level frameworks
• Tools for constructing scientific workflow
• Create virtual venues for collaborative work
• Portal a science-oriented PSE, typically with a
  web browser interface, that allows scientists to
  compose and run distributed applications, or to
  access and analyze distributed data.

40
Middleware Technology to Support Science
PortalsDennis Gannon, Randall Bramley, Indiana U.

• Novel Ideas
• A Science Portal that makes it easy to access
  Grid-based scientific applications from simple
  desktop web tools
• Organized as a set of Active Notebooks with web
  forms to launch and control the application, as
  well as histories
• Execution is governed by a grid script for that
  notebook
• Impacts
• Encapsulating Grid apps into science portals
  will make grid-based solutions easier to build
  and available to more scientists
• Using component composition of application
  services provides the first simple model for Grid
  application programming

41
SciDAC Commodity Grid (CoG) KitsGregor von
Laszewski, ANL Keith Jackson, LBNL

• Novel Ideas
• Common set of reusable components for accessing
  Grid services
• Support rapid development of Science Portals,
  Problem Solving Environments, and applications
  that access Grid resources
• Web Services that access underlying Grid
  services.
• Impact
• Allow use of Grid services from higher-level
  frameworks
• Easier development of advanced Grid services
• Easier and more rapid application development
• Encourage code reuse, and reuse of Web Services

42
SciDAC Commodity Grid (CoG) KitsGregor von
Laszewski, ANL Keith Jackson, LBNL

• Novel Ideas
• Common set of reusable components for accessing
  Grid services
• Support rapid development of Science Portals,
  Problem Solving Environments, and applications
  that access Grid resources
• Web Services that access underlying Grid
  services.
• Impact
• Allow use of Grid services from higher-level
  frameworks
• Easier development of advanced Grid services
• Easier and more rapid application development
• Encourage code reuse, and reuse of Web Services

43
Pervasive Collaborative Computing EnvironmentDeb
Agarwal, LBNL Miron Livny, U Wisconsin

• Novel Ideas
• Collaboration tools that enable connectivity and
  collaboration on a day-by-day basis
• Workflow tools that enable coordination of Grid
  computing processes and human tasks
• Support the continuum of collaborative
  interaction
• Impact
• A persistent operating environment that
  facilitates day-to-day operations within
  collaborations
• Natural collaboration capabilities for
  computation-based problems
• Building blocks for typical problems in
  scientific collaborations

44
Middleware to Support Group to Group
CollaborationRick Stevens, ANL

• Novel Ideas
• Peer-to-peer Virtual Venues servers to enable
  worldwide, secure virtual communities via
  high-end collaboration envts
• Collaborative work sharing beyond simple
  application sharing
• High end visualization envts integrated into
  collaborative spaces
• Methods of asynchronous collaboration
• Impact
• Wide-spread deployment and use to further
  scientific inquiry
• Advances in our understanding of the effects of
  distance based collaboration environments on
  group dynamics
• Extending asynchronous collaboration capabilities
  to embrace all types of data streams exchanged in
  a collaboration

45
How do we get some of these things?Can we
influence the direction of projects?

• Feedback is essential. Middleware PIs want to
  interact with you !
• All projects have dissemination plans.
• Learn project details at tomorrows Poster
  Session.
• Keep the dialogue going.

46
What collaboratory and middleware tools can we
use today?

• To Work Together
• Conferencing Access Grid, H.323, ISDN,
  NetMeeting, MSN Messenger, vic/vat, ImmersaDesk
• eMail Lists majordomo, mailman
• Shared Documents Web, NFS, ELN, EN, Notes
• Shared Display NetMeeting, Access Grid, vnc,
  SameTime
• Code Repository cvs
• To Build Distributed Applications
• Grid Services Globus, Condor, Legion, Harness,
  Cactus, CoG
• Authentication Certificates Netscape, Akenti,
  Globus

47
(No Transcript)
48
(No Transcript)
49
(No Transcript)