OSG Baseline Services

1
OSG Baseline Services

• In my talk Id like to discuss two questions
• What capabilities are we aiming for in 2005/6?
• How do we introduce new services into the OSG?

2
Guidance for Capabilities - take a wish list of
the present

• Principles and paths to deployment are guided by
  essential needs of the participating VOs
• Example list
• Ability to store, serve, catalog, manage and
  discover collaboration-wide datasets on a very
  large scale
• Ability to access opportunistically non-dedicated
  resources
• Ability to host VO managed services and agents on
  gatekeeper hosts

3
with hard lessons from the past

• Protect grid services
• that are vulnerable in a multi-VO environment
• Managed data transfer services required
• since data staging is most likely point of
  failure
• Policy-based authorization infrastructure
• to distinguish user roles within a VO
• Delay binding jobs to resources to last possible
  moment
• to optimize utilization
• Robustness and reliability required
• to keep operating costs low

4
and performance targets for the (near!) future.

• Submission of collections of O(1000) jobs should
  happen within a few seconds. It is expected that
  even a typical data analysis task will translate
  into submission of O(1000) jobs.
• WMS needs to be able to keep all available
  resources busy.
• Overall reliability must be such that task
  completion is generally guaranteed within less
  than 3 retries even for large tasks. For late
  2007 this implies gt95 success rate per job.

5
Example ATLAS Production System
DDMS
Requirements for VO and Core Services
WMS
6
..introduces challenging distributed data
management issues

• A number of catalogs in play - standardize on an
  interface
• Require traffic shaping and load balancing for
  I/O (SRM-dCache deployed on US Tier2 Centers)
• IO space resources utilized based on VO
  policies
• Site-level catalogs for managed Tier2 storage
  elements
• Reliable transfer agents used also by WMS

7
Managing Persistent Services

• VO-owned and dedicated sites will allow running
  of persistent VO specific services and agents
  for cataloging, space management, replication,
  web caching, and WMS related services
• Non-dedicated sites can be managed with VO
  services running at another site
• Or dynamically deploying VO services agents

8
Portal Partitions Edge Services

• Strategy for multiple VOs partition resources
  into VO-managed and shared
• Provision for hosting persistent Guest VO
  services agents

9
Edge Services

• Need to support VO specific agents and services
  on leased gateways
• Typical
• Need a local MySQL database
• Access to port 80 (or generally, a port range)
• Will give I/O requirements in advance
• Need this for a time gtgt job execution time

10
Example Edge Service Scalable Remote Data Access

• ATLAS reconstruction and analysis jobs require
  access to remote database servers at CERN, and
  elsewhere.
• Presents additional traffic on network, long
  latencies for remote sites bottleneck at
  central DB.
• Suggest use of local mechanisms, such was web
  proxy caches, to minimize this impact

11
Summary of OSG 0.4 targets (end 2005)

• GT4 gridftp already deployed move to GT4 gram
• Managed computing elements for multi-VO
• Edge services framework providing capabilities,
  VO-managed late binding of job-to-resources
• Job sandbox inspection, globally
• Policy and trust infrastructure
• Data location services and transfer agents
  dynamically deployed via edge services
• Site catalogs, providing VO space management

12
New Services in OSG - how?

• Requirements and schedule are determined with the
  OSG deployment activity
• Architectural coherence will maintained through
  participation with the blueprint group
• Integrate middleware services from technology
  providers targeted for the OSG
• Provide testbed for evaluation and testing of new
  services and applications
• Test and exercise installation and distribution
  methods
• Provide feedback to service providers and VO
  application developers
• Prepare release candidates for provisioning.

13
Service Readiness and Integration Plans

• Service proponents come to the integration
  testbed with an appropriately scoped
  functionality and integration plan
• Purpose, scope
• Service Description
• Packaging Description
• Dependencies resources and services needed
• Test use cases identified
• Testing tools clients, harness metrics for
  success clearly defined
• Effort to contribute to the OSG-IVC and schedule
• Links to appropriate documentation, WSDL, etc

14
Path for New Services in OSG
OSG Integration Activity
Readiness plan Effort Resources
Readiness plan adopted
VO Application Software Installation
Software packaging
OSG Deployment Activity
Service deployment
OSG Operations-Provisioning Activity
Release Candidate
Application validation
Middleware Interoperability
Functionality Scalability Tests
feedback
Metrics Certification
Release Description
15
OSG Integration Testbed Layout
OSG Integration Testbed
VO contributed
Service platform
Stable Production Release
Integration Release
Resources enter and leave as necessary
applications, test harness, clients
16
Deployed ITB
17
Validation CMS-MOP and ATLAS-Capone
18
Validation Jobs on ITB 0.1.5
19
Validation GT4 GridFTP
http//osg.ivdgl.org/twiki/bin/view/Integration/Gr
idFTP
20
Conclusions

• OSG driven by VO requirements, core capabilities,
  and principles for guidance
• Capabilities for next major release to introduce
  flexibility for maturing middleware and VO boxes
  for delegated responsibility, via Edge Services
• Paths for new services, validation and release
  process within a contributed, consortium model is
  working reasonably well so far
• Process is leading to reliable, core computing
  substrate, with VO flexibility