Main Sphinx Design Concepts

1
Main Sphinx Design Concepts

• There are two primary design components which
  comprise Sphinx
• The Database Warehouse
• The Control Process
• The Database Warehouse is the information depot
  for all of Sphinx
• All information is stored/retrieved via the
  database (or possibly multiple databases)
• There is no direct communication between
  different internal Sphinx modules
• The API for all internal modules is SQL along
  with the database schema
• The Control Process is the main point of activity
• It is a loop which calls different Sphinx modules
• All modules in the control process operate on
  stateful entities in the database, changing the
  state of an entity from one to another state
• Sphinx operates as a Finite State Machine!
• Uses data mining principles in the design
• The API for all modules is SQL along the database
  schema
• Modules mine the database for information and
  then take some action.

2
More Details

• The components are currently split (packaged) in
  the following way
• Sphinx Client
• Clarens Web Service (XML-RPC) VDT-Client
• Sends a Chimera DAX to Sphinx Server
• Receives individual concrete Jobs from Sphinx
  Server
• Submits Job ( micro DAG) to Condor-G/DAGMan
• micro DAG (for nowcould use Grid-shell or
  equivalent in the future)
• Job to create environment
• Job to move input data locally (if required)
• Job to run application
• Job to move output data (if required)
• Job to register data in RLS
• Job to clean-up environment
• Sphinx Server
• Clarens Web Service (XML-RPC)
• Receives a Chimera DAX
• Returns individual concrete Jobs (micro DAG)
  
• Provides heuristic algorithms and Satisfies
  constraints for
• policies,

3
Version 1 (current configuration)Sphinx Server
with Sphinx Submission Client
Sphinx Server
VDT Client
Sphinx Client
Data Warehouse
Message Interface
Message Interface
DAG Reducer
Multi-threaded Control Process
Job Tracker
Job Predictor
DAG Predictor
Policies
Job Submitter
Accounting
Job Admission Control
Resource Prop.
Condor-G/DAGMan
DAG Admission Control
Resource Hier.
DAG Data Repl. Planner
Resource Dist.
Distance
Job Execution Planner
Entity-Name
VDT Server Site
Entity Hier.
Resource
Job-Entity Prop.
Data Management
Job
DAG Tracker
DAG
Rel. Data
Repl. Loc. Serv.
Rel. Appl.
Information Gatherer
Job Tracking
Grid Weather
Monitoring Service
Schedule
Prediction
Message
indicates current place-holder implementation
4
Version 1 Sphinx Control Process Simple
Finite State Machine
DAG unreduced JOB unpredicted
Await Mesg.
no
yes
DAG unpredicted JOB unpredicted
DAG unfinished JOB unplanned
Reduce DAG
Plan Dag
no
no
yes
yes
DAG unpredicted JOB unaccepted
DAG unfinished JOB unsent
Predict Job
Plan Job
no
no
yes
yes
DAG unaccepted JOB unaccepted
DAG unfinished JOB remove
Predict DAG
Submit Job
no
no
yes
yes
DAG unaccepted JOB unplanned
DAG remove JOB remove
Admit Job
Remove Job
no
no
yes
yes
DAG unplanned JOB unplanned
Remove DAG
Admit DAG
no
yes
5
Examples of how planning modules can be moved
inside/outside the Sphinx Server/Client
Sphinx Server
VDT Client
Sphinx Client
Data Warehouse
Message Interface
Message Interface
DAG Reducer
Multi-threaded Control Process
Job Tracker
Job Predictor
DAG Predictor
Policies
Job Submitter
Accounting
Job Admission Control
Resource Prop.
Condor-G/DAGMan
DAG Admission Control
Resource Hier.
DAG Data Repl. Planner
Resource Dist.
Distance
Job Execution Planner
Entity-Name
VDT Server Site
Entity Hier.
Resource
Job-Entity Prop.
Data Management
Job
DAG Tracker
DAG
Rel. Data
Repl. Loc. Serv.
Rel. Appl.
Information Gatherer
Job Tracking
Grid Weather
Monitoring Service
Schedule
Prediction
Message
6
Examples of how planning modules can be moved
inside/outside the Sphinx Server/Client
Sphinx Server/VDT Client
UI Machine
Sphinx Client
Data Warehouse
Message Interface
Message Interface
DAG Reducer
Multi-threaded Control Process
Job Predictor
DAG Predictor
Policies
Accounting
Job Admission Control
Resource Prop.
DAG Admission Control
Resource Hier.
DAG Data Repl. Planner
Resource Dist.
Distance
Job Execution Planner
Entity-Name
Condor-G/DAGMan
VDT Server Site
Job Submitter
Entity Hier.
Resource
Job Tracker
Job-Entity Prop.
Data Management
Job
DAG Tracker
DAG
Rel. Data
Repl. Loc. Serv.
Rel. Appl.
Information Gatherer
Job Tracking
Grid Weather
Monitoring Service
Schedule
Prediction
Message
7
Examples of how planning modules can be moved
inside/outside the Sphinx Server/Client
Sphinx Server/VDT Client
VO Specific External Planning Service
Sphinx Client
Data Warehouse
Message Interface
Message Interface
DAG Reducer
Multi-threaded Control Process
Job Predictor
Job Execution Planner
DAG Predictor
Policies
Accounting
Job Admission Control
Resource Prop.
DAG Admission Control
Resource Hier.
DAG Data Repl. Planner
Resource Dist.
Distance
Entity-Name
Condor-G/DAGMan
VDT Server Site
Job Submitter
Entity Hier.
Resource
Job Tracker
Job-Entity Prop.
Data Management
Job
DAG Tracker
DAG
Rel. Data
Repl. Loc. Serv.
Rel. Appl.
Information Gatherer
Job Tracking
Grid Weather
Monitoring Service
Schedule
Prediction
Message