Dynamic Resource Provisioning in GRID Environments

1
Dynamic Resource Provisioning in GRID Environments

• Arkadiusz Wronski

2
Batch schedulers

• Commonly used to manage access to parallel
  computing clusters.
• Not configured to enable easy configuration of
  application-specific scheduling policies.
• Scheduling algorithms can be expensive to
  execute!
• Poor perform with large number of small tasks.

3
How to improve Batch Scheduler?

• Provisioning
• Architecture
• Implementation
• Allocation policies
• De-allocation policies
• How provisioning can change the workload
• Scheduling
• FALCON as a scheduling system

4
How to execute many tasks at once

• Data driven task graphs (DAGMan, Swift, VSD)
• Master-Worker model
• Tasks are dispatched to a cluster or GRID
• Batch scheduler used to manage cluster receive
  individual tasks, dispatch to idle processors and
  notify clients when complete

5
Disadvantages of direct dispatch

• Time required to dispatch can be large typical
  batch scheduler provides rich functionality (low
  throughput)
• The policies implemented in a particular
  instantiation may not be optimal, since batch
  schedulers may support different queues and
  policies.

6
How to improve?

• Transform the application to reduce the number of
  tasks (can be complex to do)
• Multi-level scheduling (BS allocates resources to
  which a second-level scheduler dispatches tasks)
• But authors propose
• Using adaptive Provisioner (acquire/release res.)
• Reduce average queue wait time by amortizing high
  overhead of resource allocation over the
  execution.

7
DRP system - Architecture
8
DRP system - Architecture

1. User
2. Dynamic Resource Provisioner Utilizing
   Application (i.e. Web Service such as Falcon)
3. The Provisioner
4. The Resource Manager (i.e. GRAM, Condor, PBS,
   etc)
5. Resource pool

9
DRP system - Architecture

• DRP Utilizing Application initializes Provisioner
  with set of conf. parameters (message 0)
• The state that needs to be monitored and how to
  access it
• The rules and conditions under which Provisioner
  allocates/de-allocates resources
• The location of the worker code that is specific
  to DRP UA
• The min/max number of resources it should
  allocate
• The min/max length of time resources should be
  allocated for, etc.

10
DRP system - Architecture
User submit work via message (1). DRP UA queues
up the work making it ready for processing by an
executor. The Provisioner monitors the internal
queue (message pool) and makes the decision how
many resources and for how long to allocate.
When provisioner detects the need to allocate
more resources, it contacts the Resource Manager
(2). RM is used to bootstrap the executor that
is specific to the DRP UA message (3), which
then registers with message (4) with the DRP UA
and becomes ready to process work.
11
DRP system - Architecture

1. User
2. Dynamic Resource Provisioner Utilizing
   Application (i.e. Web Service such as Falcon)
3. The Provisioner
4. The Resource Manager (i.e. GRAM, Condor, PBS,
   etc)
5. Resource pool

12
DRP system - Architecture

1. User
2. Dynamic Resource Provisioner Utilizing
   Application (i.e. Web Service such as Falcon)
3. The Provisioner
4. The Resource Manager (i.e. GRAM, Condor, PBS,
   etc)
5. Resource pool