Architectural Models for Resource Management in the Grid

1
Architectural Models for Resource Management in
the Grid

• Rajkumar Buyya
• Monash University, Australia
• Steve Chapin,
• Syracuse University, USA
• David DiNucci
• Elepar.com, USA

2
Outline

• Changes in Computing Landscape
• Resource Management Issues
• Architectural Models
• Hierarchical Resource Management
• Abstract Owner
• Market Model
• Economy Grid
• Nimrod/G Resource Broker
• Conclusions

3
Computing Power (HPC) Drivers

• Solving grand challenge applications using
  computer modeling, simulation and analysis

Aerospace
Internet Ecommerce
Life Sciences
Digital Biology
CAD/CAM
Military Applications
Military Applications
Military Applications
4
Computing Platforms
?
PERFORMANCE
Breaking Administrative Barriers

• Individual
• Group
• Department
• Campus
• State
• National
• Globe
• Inter Planet
• Universe

Desktop
SMPs or SuperComputers
Local Cluster
Global Cluster/Grid
Inter Planet Cluster/Grid ??
Enterprise Cluster/Grid
5
Towards Grid Computing
Unification of geographically distributed
resources
6
What is Grid ?

• An infrastructure that couples
• Computers PCs, workstations, clusters,
  supercomputers, laptops, notebooks,
• mobile devices, PDA, etc
• Software e.g., ASPs renting expensive special
  purpose applications on demand
• Catalogued data and databases e.g. transparent
  access to human genome database
• Special devices e.g., radio telescope
  SETI_at_Home searching for life in galaxy,
  Austrophysics_at_Swinburne for pulsars)
• People/collaborators.
• offers dependable, consistent, pervasive
  access to resources.

7
A Example Grid Infrastructure
8
Sources of Complexity in Grid Resource Management

• No single administrative control.
• No single ownership policy
• Each resource owner has their own policies or
  scheduling mechanisms
• Users must honour them (particularly external
  Grid users).
• Heterogeneity of resources.
• Dynamic availability may appear and disappear

9
Sources of Complexity in Grid Resource Management

• Unreliable resource disappear from view!
• No uniform cost model - varies from one users
  resource to another and from time of day.
• No single access mechanism Web, custom
  interfaces, command line

10
Grid Resource Management Issues

• Authentication (once).
• Specify (code, resources, etc.).
• Discover resources.
• Negotiate authorization, acceptable use, Cost,
  etc.
• Acquire resources.
• Schedule Jobs.
• Initiate computation.
• Steer computation.
• Access remote data-sets.
• Collaborate with results.
• Account for usage.

• Discover resources.
• Negotiate authorisation,
• acceptable use, Cost, etc.
• Acquire resources.
• Schedule jobs.
• Initiate computation.
• Steer computation.

Domain 1
Domain 2
Ack Globus..
11
Architectural Models
12
Hierarchical Resource Management
Access/Admission Control Agent
User
Grid Information Service
Persistent Job Control Agent
Connection Cloud
Monitor
Local Scheduler
Deployment Agent
Domain Resource Manager or Control Agent
Control Domain
- Task
Resource
13
Who owns the GRID?
Talk to people
Powerappliances
Use GRID resources
I want to
My interface is
I arrange serviceand payments with a (may be
many choices)
Abstract Owner (AO)
Phone co.
Electric co.
HPC Networks Instruments People
Generators Power lines Transformers
Antennae Cable/fiber Switches
But resources I getmay belong to others
14
AO is owner or broker
User

• User negotiates with AO through order window
• That AO may own some resources, and/or it may
  broker with other AOs for those resources
• After negotiation, resources are delivered
  through pickup window

Requests
Resources
Order Window
Pickup Window
AO
Order
Pickup
Order
Pickup
Manager
ResourceManager
Delivery
Sales
AO3
Physical Resource
AO2
AO1
15
AO Resources

• Resources are objects
• Classes are
• Instrument
• Data source, sink, transform
• e.g. programs, people, files, data collection
  devices
• Channel
• Moves data among instruments
• Complexes of above
• Attributes define sizes, times, connections, etc.

Instrument (File)
Instrument (Program)
Channels
Instrument (File)
Instrument (Program)
Instrument (Telescope)
Instrument (Person)
16
Negotiating with an AO
Make dummy resource (with attributes set to
constants, variables, or dont care) bid
delivery plan variable constraints
Pick one, Try again, Or give up
Assign tasksto resource,use, relinquish
User
Perhapslater...
Delivery Window
Order Window
Resource candidates (values for
variables/attributes asking price for each)
AO
Resource
17
Many Testbeds ? who pays ?
GUSTO
EcoGrid
Legion Testbed
NASA IPG
18
Testbeds so far -- observations

• Who contributed to resources why ?
• Volunteers for fun, challenge, fame, public good
  like SETI_at_Home distributed.net projects.
• Collaborators sharing resources while developing
  new technologies of common interest Globus,
  Legion, Ecogrid.
• How long ?
• Short duration GUSTO decommissioned.
• What do we need ? Grid Marketplace!
• Regulates demand and supply, offers incentive for
  being players, simple, scalable solution,
  quasi-deterministic proven model in real-world.

19
Users in Grid Economy Strategy

• Grid Consumers
• Execute jobs for solving varying problem size and
  complexity
• Benefit by selecting and aggregating resources
  wisely
• Tradeoff timeframe and cost
• Strategy minimise expenses
• Grid Providers
• Contribute idle resource for executing consumer
  jobs
• Benefit by maximizing resource utilisation
• Tradeoff local requirements market opportunity
• Strategy maximise returns on services

20
Building of a Economy Grid brokerage system..
Who pays for that ???
Users!
Foundation for the Grid Economy
21
Grid Architecture for Computational Economy
Information Server(s)
Grid Market Services
Sign-on
Health Monitor
Info ?
Grid Node N

Grid Explorer

Application
Secure
Job Control Agent
Grid Node1
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid User
Grid Resource Broker

R1
R2
Rm
Grid Middleware Services
Grid Service Providers
22
Economic Models for Trading

• Commodity Market Model
• Posted Prices Models
• Bargaining Model
• Tendering (Contract Net) Model
• Auction Model
• English, first-price sealed-bid, second-price
  sealded-bid (Vickrey), and Dutch.
• Proportional Resource Sharing Model
• Shareholder Model
• Partnership Model

23
Economy Grid Globus GRACE
Applications
Grid Apps.

Science
Engineering
Commerce
Portals
ActiveSheet
High-level Services and Tools
GlobusView
Grid Status
Grid Tools
DUROC
globusrun
MPI-G
Nimrod/G
MPI-IO
CC
Core Services
Heartbeat Monitor
Nexus
GRACE-TS
GRAM
Grid Middleware
Globus Security Interface
GASS
DUROC
MDS
GARA
GBank
GMD
Grid Fabric
Local Services
GRD
QBank
JVM
Condor
TCP
UDP
eCash
LSF
PBS
Solaris
Irix
Linux
24
What is Nimrod/G ?

• A resource broker for managing and steering task
  farming (parametric sweep) applications on
  computational Grids based on deadline and
  computational economy.
• Key Features
• A single window to manage control experiment
• Resource Discovery
• Trade for Resources
• Scheduling
• Steering data management
• It allows to study the behaviour of some of the
  output variables against a range of different
  input scenarios.

25
Nimrod/G Grid Broker Architecture
Nimrod/G Client
Nimrod/G Client
Nimrod/G Client
Nimrod/G Engine
Schedule Advisor
Trading Manager
Grid Store
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS
Globus,Legion, Condor-g,, Ninf,etc.
GE GIS
Grid Information Server(s)
RM TS
RM TS
RM TS
G
C
L
G
Legion enabled node.
Globus enabled node.
L
C
RM Local Resource Manager, TS Trade Server
Condor enabled node.
26
A Nimrod/G Client
Deadline
Legion hosts
Globus Hosts
Bezek is in both Globus and Legion Domains
27
User Requirements Deadline/Budget
28
Global Economy Grid
Australia
North America
Monash Uni.
ANL SGI/Sun/SP2 USC-ISI SGI UVa Linux
Cluster Manitoba Cluster
Nimrod/G
Linux cluster
GlobusLegion Condor/G
Globus/Legion GRACE_TS
Solaris WS
Internet
Europe
ZIB/FUB T3E/Mosix Cardiff Sun E6500 Paderborn
HPCLine Lecce Compaq SC CNR Cluster CERN
Cluster
Asia/Japan
Tokyo I-Tech. ETL, Tuskuba
Linux cluster
Globus GRACE_TS
Globus GRACE_TS
29
(No Transcript)
30
Conclusions

• Proposed three models for Grid resource
  management architecture
• Hierarchical, AO, Market-model
• The future systems are likely follow a model that
  combines all these models.
• The future computing (HPC) infrastructure is
  going to be a Grid of Clusters.
• Peer-to-Peer/Grid has already become a darling of
  venture capitalists.
• The impact of Grid on 21st century economy will
  be the same as electricity on 20th century
  economy.