Scheduling

1
Scheduling Resource ManagementWorking Group

• Jennifer Schopf Bill Nitzberg, co-chairs
• www.cs.nwu.edu/jms/sched-wg/
• sched-wg_at_gridforum.org
• Grid Forum 5 Meeting
• Boston, MA
• Oct 15 - 18, 2000

2
Scheduling Working GroupObjectives Progress
3
High-level OverviewSolve Grid Resource
Management

• Who? -- Developers
• What? -- Agreements / standards
• Capabilities, general protocols, APIs
• Why? -- Interoperability
• Reserving, allocating, using resources
• Managing resources (owners pt-of-view)
• Support co-scheduling diverse resources
• Enable "better" "use of" resources

4
Charter

• Look at what is done today, gather
  requirements
• ...refining protocols, interactions, etc.
• ...work to standardize APIs
• Current Focus Areas
• Advance reservations
• Super scheduling
• Token definition
• information proper nouns, semantics,
  representation

5
Progress

• Grid Query and Reservation Interface, SchedWD2.2,
  Q. Snell, D. Jackson, M. Clement
• Generic Grid Resource Description, SchedWD9, K.
  Czajkowski, G. von Laszewski, J. Gehring
• Security Requirements of the Scheduling Working
  Group, SchedWD6.9, K. Jackson V. Sander
• Ten Steps for Superscheduling, SchedWD8, J.
  Schopf, Q. Snell, K. Czajkowski
• (Advance Reservation API, SchedWD 4, 5, 7, A.
  Roy, V. Sander, Q. Snell, J. Gehring)

6
Scheduling Working GroupA Brief History
7
Advance Reservation Co-Scheduling Workshop,
May 1999

• Defined reservation
• Resource start end duration
• Enumerated desired capabilities
• de-coupled from job submission
• unique printable reservation ID
• query/response - returns list of available slots
• hard and soft reservations
• Enumerated harder stuff to put off til later,
  e.g., guarantee, cost model

8
Grid Forum 1 (NASA Ames)June 1999

• Initial Charter
• Solve Grid Resource Management
• Three focus areas
• Advance reservations
• Super scheduling
• Resource specification (semantics tokens)

9
Grid Forum 2 (Northwestern)October 1999

• Refined charter
• Requested
• lists of tokens from different groups
• architecture pictures of existing systems
• Discussed What is X?
• e.g., job, scheduler

10
Grid Forum 3 (UCSD)March 2000

• Adopted charter refocused
• Decided not to work on architecture
• Developed Super-scheduler Model (10 steps)
• Gave overviews of advance reservation systems
  (GARA, Maui, PBS, LSF)
• Commitments to draft several SchedRFCs

11
Grid Forum 4 (Microsoft)July 2000

• Changed Sched RFC to Sched Working Document
• Revised working document drafts
• Query Interface
• Resource Acquisition Steps
• Security Requirements
• Advance Reservation API
• Scheduling Information
• Suggested new working documents
• 10 Steps Run a job API

12
Grid Forum 5 AgendaOctober 2000

• Now Working Group Update
• Mon 1300-1500 Generic Grid Resource Description
  Ten Steps for Superscheduling
• Mon 1500-1630 Security Requirements of the
  Scheduling Working Group
• Mon 1630-1800 Grid Query and Reservation
  Interface
• Tue 930-1100 Commercial Grid Systems Track
• Tue 1500-1630 Vendors, Standards, and Low
  Hanging Fruit

13
Process

• Review working document (lt 20 minutes)
• Focus on understanding the document (revisions)
  rather than correcting it
• Gather discussion items during/after review
• Prioritize discuss each item
• Next steps (last 5 minutes)
• Revised documents posted to web site within 2
  weeks.

14
Generic Grid Resource Description

• Goal Standardize scheduling information -- the
  tokens or labels, semantics, and their
  representation

15
Ten Steps for Superscheduling

• Goal List the basic steps and capabilities
  involved in resource reservation, acquisition,
  and use

16
Security Requirements of the Scheduling Working
Group

• Goal Ensure interoperability between GF security
  standards and GF scheduling standards

17
Grid Query Interface

• Goal Refine interface to a scheduler to answer
  the question(s)
• When will my job start?
• What times can a reservation be guaranteed?

18
Advance Reservation API

• Goal API for advance reservations
• Merge existing interfaces...
• CCS
• GARA
• Legion
• Maui
• PBS

19
Additional Sessions

• Commercial Grid Systems Track
• Vendors, Standards, and Low Hanging Fruit
• There are at least 3 standards efforts
• Are we working on the right standards?
• Coordinate / merge / move efforts

20
Questions?

• Possible change 100p - 300p
• Primary authors for Resource Description 10
  Steps are unavailable...

21
Generic Grid Resource Description

• Notes

22
Generic Grid Resource Description

• Goals of this document
• 1. Want same structured language for allocation
  reservation requests
• 2. Point is not the syntax, but the capabilities
  that the syntax should support
• What could we talk about now?
• Is this the right general direction?
• List of resources?
• List of capabilities that the syntax must
  support?
• What is the target of the language? Who is
  communicating?
• Sender (requester) is seeking resources from the
  receiver (provider) provider allocates/reserves
  resources
• Publisher/subscriber of information about
  resources
• Declarative vs. Procedural?
• No clear concensus (or agreement on definitions)

23
GGRD Notes

• Requirements of language -- Ability to express
• constraints in users requests in the resource
  base
• Fault domain
• Workflow
• May be overloading language by requiring the
  distinction between description and instantiation
• What about GIS GOS?
• An entity-relationship model might be better
  suited to our needs
• Place constraints on each entity and relation
• Need to deal with hard constraints, soft
  constraints, and priorities (or ranking)
• Should we be looking at UML?
• In literature for specifying automatic testing,
  there are formalisms for describing time
  constraints.

24
GGRD Notes

• We are quickly increasing the domain of this
  language
• What should the scope be?
• Perhaps the language should capture everything
  from the rough cut through the final allocation
  (and every step along the way)
• May want to focus on a smaller scope at first
• Proposal have document reflect the smallest
  scope possible to stimulate/focus discussion for
  next time.
• What needs to be done with this documents
• More clearly define the target(s) of this
  language
• Focus this document on, and describe more
  explicitly, the requirements of the language (or
  the capabilities that the language must support)
• Then move onto language and syntax

25
Generic Grid Resource DescriptionK. Czajkowski,
(G. von Laszewski), (J. Gehring)

• Goal Standardize scheduling information -- the
  tokens or labels, semantics, and their
  representation
• Confusion over focus of the document
• Meant to list capabilities that the language
  should support (not the syntax)
• Next Steps
• Refocus on requirements/capabilities of language,
  then move onto language syntax

26
Ten Steps for Superscheduling

• Notes

27
Ten Steps for SuperschedulingJ. Schopf, (Q.
Snell), (K. Czajkowski)

• Goal List the basic steps and capabilities
  involved in resource reservation, acquisition,
  and use
• No major issues raised
• two minor editorial changes proposed
• Next Steps
• Expand on last section
• Should be ready for presentation as proposed GF
  Document at the next GF

28
Security Requirements of the Scheduling Working
Group

• Notes

29
Security Notes

• Suggestion make a list of security requirement
  for scheduling group, and then hand it over to
  the security group
• Should this doc. be merged into the overall
  Security Requirements document?
• Request for a recommendation from the GF Security
  group as to which security stuff to use (e.g.,
  GSI?)
• maybe a best practices
• Missing from the document?
• need to authenticate a user in a specific way
  (e.g., user has access because they have a
  contract with us) -- need mechanisms to handle
  legacy policies

30
Security Notes

• Perhaps we need to begin policy discussions among
  different agencies (countries, etc.) to permit
  Grid computing (from the security perspective)
• Isnt this happening in the CA discussions?
• Plus, theres not really a need right now for DoE
  to interoperate with DoD
• Separate out issues (policy mechanism) and
  focus on the information needed to do X
• (General concensus is that there are really only
  two choices for authentication PKI and Kerberos)
• Goals of this document bring out issues in
  scheduling to the security group (and vice
  versa).
• Proposal that we should re-visit
  scheduling/security from time to time

31
Security Requirements of the Scheduling Working
GroupK. Jackson V. Sander

• Goal Ensure interoperability between GF security
  standards and GF scheduling standards
• No issues raised
• Next steps
• To be merged into the Scenarios Security document
• Vauge non-commital commitment to look into a best
  practices paper

32
Grid Query Interface

• Notes

33
Grid Query Interface Q. Snell, (D. Jackson), M.
Clement

• Goal Refine interface to a scheduler to answer
• When will my job start?
• What times can a reservation be guaranteed?
• No major issues raised proposed APIs
• QUERY UserID Duration ResourceList OtherInfo
• RESQUERY UserID TimeSpec ResourceList OtherInfo
• Next steps
• Bind generic API to C
• Proposals expected from D. Jackson, P. Foley,
  B. Nitzberg
• Proposed APIs after QUERY
• Reservation API (in progress)
• Binding a job to a reservation, cancellation,
  monitoring

34
Monitoring Discussion(focus on active
(cancellable) jobs)

• Requires both query event (publish/subscribe)
  model
• Failure semantics / recovery
• If the metascheduler/scheduler dies, should jobs
  still get events?
• Random set of stuff were interested in
• API for a job to be able to post its
  job-specific state
• Get state of all jobs with tag X
• Get the original request for job J
• Get the list of allocated resources for job J
• Get environment for job J
• Get info on staging files
• Set thresholds and get notifications when
  thresholds are exceeded
• Register interest in events (publish/subscribe)
• Proposal expected from anand_at_virginia.edu
• should coordinate w/Performance Group documents

35
Vendors, Standards, and Low Hanging Fruit

• Notes

36
Vendors, Standards, and Low Hanging Fruit

• This group has been around for 1 year
• Two questions
• Are we focused on the right stuff
• What about the other standards efforts

37
Current Effortse.g., people are expending time
on these

• Query, API, Q. Snell, D. Jackson, M. Clement
• Generic Grid Resource Description,
  Syntax/Language, K. Czajkowski, G. von Laszewski,
  J. Gehring
• Ten Steps, Architecture/Framework, J. Schopf, Q.
  Snell, K. Czajkowski
• Advance Reservation, API, A. Roy, V. Sander, Q.
  Snell, J. Gehring

38
Is There Consensus to Work on Other Stuff?

• Advance Reservation protocol?
• API was easier to agree on among groups writing
  advance reservation systems --- protocols were
  thought to be harder (to agree on)
• Tokens? (Uwe.Schwiegelshohn_at_udo.edu
  haupt_at_erc.msstate.edu)
• Will the GGRD document include these?
• Scheduling Landscape document (P. Foley)
• Describe the different entries in the scheduling
  arena marketing document representing all the
  different schedulers in this room -- framework
  on which to test scenarios

39
What About theOther Standards Efforts

• Currently
• Grid Forum
• Intel Peer-to-Peer Working Group (PtPWG)
• New Productivity Initiative (NPI)
• Broadband (?)

40
Progress

• Teleconferences (every other week)
• Wed - Noon East Coast Time (9am Pacific, 6pm in
  Germany)
• Documents should be distributed lt 1 week before
• Nov 15 Ten Steps, Architecture/Framework, J.
  Schopf, Q. Snell, K. Czajkowski
• Nov 29 Advance Reservation, API, A. Roy, V.
  Sander, Q. Snell, J. Gehring
• Dec 13 Generic Grid Resource Description,
  Syntax/Language, K. Czajkowski, G. von Laszewski,
  J. Gehring
• Jan 10 Query, API, Q. Snell, D. Jackson, M.
  Clement