MonALISA

1
Iosif Legrand California Institute of
Technology
2
Distributed Dynamic Services Architecture

• MonALISA is a Distributed, Dynamic Service
  Framework
• Hierarchical structure of loosely coupled
  services which are independent autonomous
  entities able to cooperate using a dynamic set of
  proxies or self describing protocols.
• They need a dynamic registration and discovery
  subscription mechanism. For an effective use of
  distributed resources, these services should
  provide adaptability and self-organization
  (aggregation and hierarchical orchestration)
• Flexible access to complex monitoring
  information.

3
Distributed Object Systems CORBA , RMI, DCOM
Traditional Distributed Object Models (CORBA,
DCOM)
The Stub is linked to the Client. The Client must
know about the service from the beginning and
needs the right stub for it
CLIENT
Server
IDL Compiler
The Server and the client code must be created
together !!
4
Web Services WSDL/SOAP
SOAP
CLIENT
Server
The client can dynamically generate the data
structures and the interfaces for using remote
objects based on WSDL
Platform independent
5
Distributed Services

• Act as a true dynamic service and provide
  the necessary functionally to be used by any
  other services that require such information
  (Jini, interface to WSDL / SOAP)
• mechanism to dynamically discover all the
  Service Units"
• remote event notification for changes in the any
  system
• lease mechanism for each registered unit

6
JINI Network Services
A Service Registers with at least one Lookup
Service using the same ID. It provides
information about its functionality and the URL
addressed from where interested clients may get
the dynamic code to use it. The Service must ask
each Lookup Service for a lease and periodically
renew it. If a Service fails to renew the
lease, it is removed form the Lookup Service
Directory. When problems are solved, it can
re-register.
jar
jar
Web
Web
Server
Server
CLIENT
Publish the
Publish the
Interface jar
Interface jar
SP
Register
Register with ID
Service ID
Service
Ask for a lease Get a lease for DT
The lease mechanism allows the Lookup Service to
keep an up to date directory of services and
correctly handle network problems.
7
Monitoring Data Collection
Dynamic Thread Pool
Other tools (Ganglia, MRT)
PULL
SNMP get walk rsh ssh remote
scripts End-To-End measurements
Farm Monitor
Configuration Control
Trap Listener
PUSH snmp trap
WEB Server
Dynamic loading of modules or agents
Trap Agent (ucd snmp) perl
8
The Muti-Threaded Execution Architecture

• Each request is done in an independent thread
• A slow agent / busy node does not perturb the
  measurements of an entire system
• Ex Monitor 300 nodes _at_ 30 seconds interval
  ?10-15 Threads are running in parallel

9
Data Handling

• Data Model
• Configuration Farm , Function (Cluster),
  Node, Module
• Monitored Values
  
  Parameter, Value
• (Automatic) Mapping of the Data Model in
• XML, SQL, SOAP,
• Configuration Results objects are store in a DB
• (dynamically configurable for InstanDB, Postgres,
  MySQl, Oracle )
• Subscription to results objects matching a
  template / predicate
• Clients can load filter objects into the Data
  Cache service and generate any derived (or
  aggregate) data structures and register to
  receive them.
• Monitored parameters have a life time

TIME
10
Service Monitor UNIT Data Handling
Lookup Service
Lookup Service
Monitor Data Stores
Client (other service) Web client
WEB Service WSDL SOAP
Farm Monitor
Data Cache Service DB
Discovery
Registration
Client (other service) Java
data
McKoi DB MySQL
Predicates Agents
Other tools
Configuration Control (SSL)
UDP
MySQL
Predicates Agents
User defined loadable Modules to write /sent data
MDS
11
Registration / Discovery / Remote Notification
Discovery
Registration
Client (other service)
Lookup Service
Services Proxy Multiplexer
Data Filters Agents
Services Proxy Multiplexer
Lookup Service
Client (other service)
12
Secure Automatic Update Mechanismfor Services,
Clients Embedded Applications
Lookup Service
MonaLisa Service
Web Server Sign Distribution
Discovery
download
Update Signal SSL

• All running services are update using the
  discovery mechanism
• At startup each service check if it an update is
  done at a set of Web Servers
• Clients use the Web Start mechanism

download
Admin Client
Update Signal SSL
MonaLisa Service
Lookup Service
EMBEDDED APPLICATIONS Monitor, Control
Execution, Update
13
Global Client for Farms and Network
Connectivity
DataTAG
-
-
14
Real-time Data for Large Systemslxshare
cluster at cern 600 ndoes
15
Access to historical and real-time values
Past values are presented and the GUI remains a
registered listener and the new vales are added
Real Time Histograms for various parameters
16
Mobile Agents and Filters
Simple Global Load filter agent
From FNAL to all
Maximum Flow Data Replication Path Agent Deployed
to each RC and evaluates the best path for
real-time data replication
From CERN to all
17
Pseudo Clients Dedicated Repositories
18
MonALISA repositories
19
VRVS Architecture
pub
cal- tech
cor- nell
funet
Reflectors are hosts that interconnect users by
permanent IP tunnels.
vrvs 5
star- light
vrvs us
vrvs eu
The active IP tunnels must be selected so that
there is no cycle formed.
usf
sinica
inet 2
Tree
usp
The selection is made according to the assumed
network links performance.
kek
triumf
20
Minimum Spanning Tree Algorithms
Finding a tree T that contains all the
vertices of a graph G spanning tree and
has the least total weight over all such trees
minimum-spanning tree (MST)
Input A weighted connected graph G (V,E) with
n vertices and m edges Output A
minimum- spanning tree T
21
Monitoring VRVS Reflectors
22
Monitoring VRVS Reflectors
23
Global Client / Dynamic Discovery
24
SUMMARY

• MonALISA is able to dynamically discover all the
  Service Units" used by a community and through
  the remote event notification mechanism keeps an
  update state for the entire system
• Automatic secure code update (services,
  embedded applications and clients) .
• Dynamic configuration for services. Secure Admin
  interface.
• Access to aggregate farm values and all the
  details for each node
• Selected real time / historical data for any
  subscribed listeners
• Active filter agents to process the data and
  provided dedicated / customized information to
  other services or clients.
• Mobile Agents for decision support and global
  optimization.
• Dynamic proxies and WSDL WAP pages for
  services.
• Embedded SNMP support and interfaces with other
  tools ( LSF, PBS, Ganglia, Hawkeye, IEPM-BW,
  MRTG, RRD, Generic Web Services)
• Dedicated pseudo-clients for repository, WAP
  access or decision making units
• It proved to be a stable and reliable distributed
  service system. It is currently running at 120
  sites
• http//monalisa.cacr.caltech.edu