Decentralised Middleware and Workflow Enactment

1
Decentralised Middleware and Workflow Enactment

• Daniel Goodman
• Daniel.Goodman_at_oerc.ox.ac.uk

2
Outline

• Why do this?
• Topology of the middleware
• Decentralised structuring of middleware
• Decentralised execution of workflows
• Conclusions

3
Why Another Middleware?

• Produced to assist with the analysis of climate
  prediction .net data.
• This data is too large to move, and partitioned
  in such a way that it is not feasible to write
  workflows to address a specific partitioning.
• Existing workflow languages and middleware fail
  to address this problem.

4
Topology of Middleware

• Middleware is constructed from three elements
• Data Store Constructs data structures from data
  stored at the resource it resides on.
• Data Processor Performs operations on data
  structures made up of a single piece, i.e. A
  Matrix or Set.
• Process Coordinator The component users
  interact with. This element adjusts the generic
  workflows to match the partitioning of the data
  and schedules their execution on the Data
  Processors.

5
Topology of Middleware
6
Addressing Resources

• All Data Structures and Functions are addressed
  by URIs.
• These URIs are unique to both the server and the
  system as a whole
• These contain the URL of the server they are
  currently residing on.

7
Locating servers

• Process Coordinators maintain lists of Process
  Coordinators and Data Processors.
• User tells the Process Coordinator the location
  of a new server.
• A call is made using a an argument from an
  unknown server.
• Periodically, and when discovering new resources,
  they synchronise with the other Process
  Coordinators in their lists.

8
Addition of a Single Server
PC
PC
PC
DP
DP
9
Merging Two Sets of Servers
PC
PC
PC
PC
DP
DP
10
Discovery of Services

• Data Processors publish XML documents on web
  servers associated with their address, describing
  the services they provide. These include
• The name of the function
• The namespace (ensures uniqueness)
• The arguments and if they are read or write
• Data Processors can return different XML
  documents for different users if they have
  different access properties
• If a function is to be removed, it is removed
  from the document before, allowing servers time
  to update their records

11
Decentralised Execution

• Workflow execution is decentralised by both
• Multiple Process Coordinators, allowing workflows
  to be invoked from a range of locations
• Once invoked, workflows are broken up and
  distributed to Data Processors.
• Data Processors then communicate with each other
• They only refer back to the process coordinator
  at the end of the execution
• Or, if there is a problem they are unable to
  overcome. i.e. The failure of a Data Processor
  they are working with

12
Example
PC
DP
DP
DP
DP
13
Conclusion

• Allows many independent institutions to construct
  and then merge resources
• Reduces the need for user interaction to
  configure middleware
• Reduces the number of workflows effected by a
  failing node
• Reduces the amount of network communication.