Ubiquitous Resource Monitoring with GridRM

1
Ubiquitous Resource Monitoring with GridRM

• Mark Baker and Garry Smith
• Distributed Systems Group
• University of Portsmouth, UK
• http//gridrm.org/

2
Outline

• Background.
• Overview.
• Architecture.
• The GridRM Portal.
• Summary.
• Further Work.

3
Background

• Lack of knowledge about the status of the
  resources in any distributed system will hamper
  strategies for optimal scheduling, allocation and
  usage.
• There is a need for a flexible framework that
  provides information about the health and status
  of Grid resources
• Gathering resource information, such as
• Compute (nodes, CPU, memory),
• Network (inter-site communications links, network
  devices),
• Sensors (specialised devices, Web cam,
  microphone),
• Software services (information services,
  schedulers).
• Need a generic system that does not need another
  local agent, but can utilise whatever exists
• SNMP, Network Weather Service, NetLogger,
  Ganglia, /proc, MDS,

4
Overview

• GridRM is resource monitoring system that
  provides
• A scalable monitoring architecture,
• A standards-based information system that is
  independent of underlying middleware,
• A means of harvesting heterogeneous data and
  presenting it in a homogeneous form,
• An abstract means of data collection,
• An extensible infrastructure that can be bound to
  diverse data sources (legacy, and emerging ones),
• A means of gathering data that can be used for a
  range of purposes such as scheduling, autonomic
  computing, verification of SLA, and creating
  higher-level knowledge

5
Architecture
6
Global Layer
7
jGMA

• Needed a lightweight implementation of the GGF
  Grid Monitoring Architecture in Java.
• Used jGMA, a Java-based version of the GMA for
  details see
• Mini-Workshop Grid Performability Modelling and
  Measurement
• Friday 3 September 2004
• 1110 - 1315 - Parallel Session 6
• Room Auditorium

8
Local Layer
9
Data Management Issues
10
Solutions Simple API

• Producing an API is fairly simple, but creating
  one that will be taken up and accepted is another
  matter.
• We using an API based on JDBC from Java.
• Example of API
• Agent Driver Interface
• forName(GridRM.sql.driver.NWSDriver)
• forName(GridRM.sql.agent.SNMPv1Driver)
• Connection Interface
• String agentURL gridrm//barney5550/GLUECE_ho
  st
• Connection con DriverManager.getConnection(agent
  URL)
• Statement Interface
• Statement stmt con.createStatement()
• ResultSet rs stmt.executeQuery(SELECT FROM
  ProcessorLoad)
• Manipulating Results
• ResultSet is another interface contains a handful
  of methods for manipulating the data returned
  from the agent.

11
Solutions XML Schema

• No single naming schema for this area at the
  moment.
• We needed something to mark-up the information
  that can be gathered by the local agents
• Static and dynamic information
• Name/IP/OS/Processor/NIC/
• CPU load/memory available/disk space/network/
• Did not want to produce our own schema, so chose
  an emerging one that is increasingly being used -
  Grid Laboratory Uniform Environment (GLUE)
  schema
• A schema that defines the attributes of computer
  system resources (CE/NE/)
• Others, CIM, UNICORE, etc..

12
GridRM Drivers and Manager

• The GridRM Driver Manager locates appropriate
  drivers for underlying agents
• Passes SQL requests to drivers and returns
  normalised results
• The Driver Manager also provides other
  functionality that is particular to GridRM such
  as configuration, caching, streaming or
  pushing/pulling data to/from clients.

13
Local Layer Use of SQL

• SQL used extensively throughout the framework.
• All resources are seen as databases and queried
  using SQL.
• Resource queries enter the framework as SQL
  syntax.
• Pluggable resource drivers are implemented as
  JDBC drivers
• Translate SQL requests into native protocol.
• Normalise results according to selected schema.
• Framework benefits from a single, flexible
  approach to resource interaction.
• Makes for a simple, extensible framework.

14
GridRM Portal

• The GridRM Portal (gridrm.org) is a demonstration
  of gateways, data sources, SQL and data
  normalisation.
• An example of the use of GridRM, particularly its
  ability to discover and utilise resource data.
• An example of a GridRM client which
• Allows the use of GridRM with no knowledge of the
  underlying technologies.
• Hides details like SQL, XML, etc
• Provides an abstraction everyone can use
  clickerity click!).

15
GridRM Portal
16
Portal Simple Queries

• Query resources across configured gateways.
• Pre-defined query options point click!
• A Web form shields the user from SQL syntax and
  other low-level details.
• An example query
• Retrieve details of all compute resources
• The user is authorised to use,
• From the selected Grid sites,
• That meet specified constraints for
• Memory, processor load, architecture.

17
Portal Simple Query Form (2)
18
Portal Simple Query Results
19
Portal Simple Query Results

• Comparative views of resource processor loads
  from the query in the previous slide
• Missing bars indicate idle machines.

20
Portal SQL Interface

• Aim Query a given data source, using SQL, for
  performance attributes, but without knowing the
  monitoring protocol it natively supports.
• The SQL commands can
• show gateways Get a list of registered gateways.
  
• use gateway Select a gateway to query DSG
  Workgroup Gateway.
• show datasources Get a list of all data sources
  registered with the gateway.
• use datasource Select a particular data source -
  homer.dsg.port.ac.uk
• show databases Retrieve a list of naming schemas
  supported by this data source driver.
• use GLUECE_host Query a host resource data using
  the GLUE Computing Element schema.

21
Portal SQL Interface

• show tables Observe the way the naming schema
  partitions resource data.
• desc architecture Describe what an architecture
  table will provide.
• select from Architecture Read the
  architecture type for this data source.
• select Last5min, Last15min from ProcessorLoad
  Read the 5 and 15 minute load averages.

22
Portal SQL Interface

• show gateways Get a list of registered gateways
• We need to find the list of gateways currently
  available to us.
• Enter show gateways into the command line

23
Portal SQL Interface

• The list of gateways is returned.
• Select a gateway to query with the use gateway
  syntax
• We enter the DSG gateway.

24
Portal SQL Interface

• The show datasources command returns a list of
  all data sources registered with the gateway

• The gateway is selected and we now want to see
  data sources

• We choose the holly data source

25
Portal SQL Interface

• We decide to go ahead and query host resource
  data using the GLUE Computing Element schema.
• Select the database use GLUECE_host.
• Display the tables available to us show tables
  (on the chosen GLUECE_host database).

• The translation schema partitions the retrieved
  data into tables.
• We can select all fields or individual named
  fields in each table.

26
Portal SQL Interface

• Get the table definition desc Architecture

27
Portal SQL Interface

• Query the resource driver for Architecture and
  CPU load averages for last 5 and 15 minute
• select from Architecture
• select Last5min, Last15min from ProcessorLoad

28
Summary

• Heterogeneous information returned from a diverse
  range of possible data sources.
• Need to harvest data into a homogeneous form
• Hide underlying complexity from clients.
• Provide data in a format that meets a clients
  requirements.
• Combine legacy resources with modern cluster and
  Grid information servers to provide
• An over-arching Grid information system.
• Independent of particular middleware and
  services.
• GridRM promotes homogeneity through
• JDBC-like data source driver,
• Standard SQL syntax,
• The GLUE naming schemas,
• Request translation and result normalisation,
• GLUE XML encoded results.

29
Future Work

• Provide an example of a job submission system
  using GridRM, several options
• Other schedulers, Condor, SGE,
• Further security
• Integrate UK e-Science certificates for resource
  access control.
• Performance and scalability testing.
• More translation schema for different resource
• DBMS, telescope, surf conditions!
• Use of portlet technologies to provide a better
  Web interface - GridSphere
• Secure interface for remote Gateway
  administration.

30
More Information

• GridRM, http//gridrm.org
• jGMA, http//dsg.port.ac.uk/projects/jGMA/
• GMA WG, http//www-didc.lbl.gov/GGF-PERF/GMA-WG/
  
• GLUE, http//www.hicb.org/glue/glue.htm
• JDBC, http//java.sun.com/products/jdbc/
• Ganglia, http//ganglia.sourceforge.net/
• SNMP, http//net-snmp.sourceforge.net/
• NetLogger, http//www-didc.lbl.gov/NetLogger/
• NWS, http//nws.cs.ucsb.edu

31
Testbed Status (1)

• Currently have 12 sites across 8 countries

32
Testbed Status (2)

• List of sites and status http//gridrm.org/testbe
  d.html