Operations Applications

1
Operations Applications

• Leigh Grundhoefer
• Indiana University

2
Operations Applications - Agenda

• Testing and verification of a Common Service
  environment, with the Integration Test Bed (ITB)
• GOC Support, and Support Center Coordination
• Problem tracking
• Information Management
• Validation of resources
• Operational Status and Job Monitoring

3
Goals of Operations

• Ensure that the production environment is usable
  for the current application base
• Coordination of software, services and people
• Continue to evolve the common service environment
  which supports multiple sciences with a
  application-friendly infrastructure.
• Management of change

4
Operations Activities
Other Grids
Security, Policy and Authentication
Registration, Verification and Monitoring
5
OSG Integration Test Bed (ITB)

• Each ITB release is an evolutional step adding,
  removing, and updating the functional set of
  services
• Provides a forum between the service developers,
  cluster administrators, and applications
  developers
• The test bed is used to identify issues of
  scalability, usability and incompatibility of
  services and service versions.

6
ITB - Starting points

• Production Release dates
• Information and Plans from Virtual Organizations,
  stakeholders, experiments
• Virtual Data Toolkit Release Schedule
• Feedback from Production Operations

7
ITB - Completion points

• No report of problems from Resource
  Administrators
• Services which were found to be problematic,
  patched or removed
• Approaching deadline of Production Release
• VOs successfully tested their applications

8
Validation of Resources

• ACDC Operation Dashboard provides detailed cyclic
  testing of all resources
• Tests based upon site-verify tool distributed
  with OSG common software
• Tests results output available per test per site
• Five possible results
• No Information
• Pass
• Fail
• Error
• Not Tested
• Resources are grouped in to three areas
  Production, Pending or Offline

9
Common Problem and Issues

• Resource disruptions
• Configuration problems
• Scheduled service periods
• Dynamic events System load, storage space and
  network outages
• Common service feature enhancements requests
• Documentation problems
• Issues which are effect a singe Virtual
  Organization
• Authorization
• Rates of failed jobs

10
OSG Support Center Model

• Use existing support infrastructure when possible
• Network Operations Center
• University Help-Desk
• Experiment based support centers
• Establish Resource or Service and the support
  center at the same time
• Make trouble reporting and interaction follow a
  logical path. (Support Centers interact.)
• Report locally. Compute Globally

11
(No Transcript)
12
OSG Community Support

• How do we support issues that fall outside the
  Support Model?
• Open Support Model
• Mailing Lists (OSG-General)
• Knowledge Base and Release Documentation
• Jabber chat room
• Weekly meetings

13
Problem and Issue Management

• GOC tickets are created by phone and web form,
  but most likely from trouble reported to
  goc_at_opensciencegrid.org or an OSG mailing list.
• GOC tickets are routed to a support center for
  action, and are discussed in detail during the
  Monday Operations teleconference.
• Support centers open tickets with the GOC for
  action outside of their support center.
• GOC will follow-up on all tickets and inform
  originator before closing a ticket.

14
OSG GOC - Support Workflow
Grid Validation Services and Reports from Users
Experiments
Developers/ Integration
15
Information Management tools

• Registration database of Resources, VOs
• Administrator, Physical Location and
  Descriptions, etc.
• Support Center contacts
• Information Catalogs of Resources, VOs
• Glue Schema and Generic Information Provider
• Publishing of resource configurations and current
  status
• ACDC operations dashboard
• Resource Administration and Release Validation
• Validation of resources for each VO

16
Production Operations Activities

• Reports
• Usage reports from OSG monitoring
• Operations reports to the OSG community
• Daily Sites Status reports to Support Centers
• Meetings
• Weekly Operations Activity
• Release-based provisioning Activity
• Weekly Support Centers Technical Group
• Weekly Documentation Activity
• Monitors for verification
• ACDC Operations Dashboard

17
GOC Conclusions

• Creates a known and usable service environment
• Allows for service validation and trouble
  reporting path
• Helps users and applications bridge the gap
  between single use and grid based resource
  utilization

18
Thank you!

• Kristy Kallback Fred L. Rob Quick Kyle Gross
  Leigh G. John Rosheck (Tim Silvers)