AstroGrid

1
AstroGrids Common Execution Architecture

• Guy Rixon, reporting on behalf of Paul Harrison
  and the other AstroGrid developers
• SC4DEVO-1, Pasadena, July 2004

2
AstroGrid

• UK national virtual-observatory project
• See cover slide for participants and funding
• Tasked to produce a VO toolkit to be reused by
  service providers
• Collective author of a SOA using web services
• Common Execution Architecture
• A major part of AstroGrid architecture
• Part of the service-oriented architecture
• A way of making and using web services
• Some important applications to data mining

3
SOA for data mining

• How can we carve up the distributed
  application into services?
• And where do we put the application logic?

4
Distributed app conservative
Desktop
Application
Application logic in here
Data-centre 2
Data-centre 1
Data transform
Data source
5
Distributed app browser-based
Desktop
Browser
Application logic in here
Data-centre 2
Data-centre 1
Data transform
Data source
6
Distributed app intragrid
Desktop
Some application logic in here Rest still in
client or in user
Browser/app
Node
Data source
Data transform
Node
Node
Node
Data-centre 2
Data-centre 1
Compute grid
7
Distributed app scripted
App logic in here
Desktop
Script
Browser/app
Workflow engine
Data source
Data transform
Data transform
Data centre 3
Data-centre 2
Data-centre 1
8
Distributed app embedded
Little app logic here
Desktop
Script
Browser
Workflow engine
Most app logic here
Data source
Data transform
Application
Data mine
Data-centre 2
Data-centre 1
9
Distributed app data grid
Desktop
Script
Browser/app
Workflow engine
Data source
Data transform
Data transform
Data centre 3
Data-centre 2
Data-centre 1
10
Issues to be addressed

• Boring, technical
• How to write apps as w/s?
• How to write w/s clients?
• How to connect apps to storage?
• How to register functions?

• Interesting, scientific
• Who has useful algorithms?
• How to match algorithms to data?
• Best place to run computations?
• How to share code?
• Provenance?
• How to share kudos?
• Is more science possible?

11
AstroGrid architecture and products

• AstroGrid bits and pieces let techies solve the
  tech problems so that scientists can concentrate
  on the science issues.
• AstroGrid supports
• Browser-based app
• Scripted app
• Data grid
• Embedded app (pre-installed exes)
• Common Execution Architecture lets all parts but
  exe for embedded app be reusable code.

12
Common Execution Architecture (1)
CEC Common Execution Connector
Web service (CEC)
MySpace
CEC configuration
Application (local exe)
Data mine
13
Common Execution Architecture (2)
Scripts stored here, as files
MySpace
Page presentation
Workflow executor (JES)
Workflow definition
Portal
14
Common execution architecture (3)
Common language throughout system allows s/w
reuse.
15
How to write an app using CEA?

• Dont write a web service!
• Write a command-line exe
• Configure a CEC to run your app
• I.e. describe parameters in the CEA language
• Or get your app configured on someone elses CEC

16
How to publish an app using CEA?

• Take the ltApplicationDefinitiongt that informs the
  CEC.
• Wrap it in a ltVOResourcegt of sub-type
  CeaApplicationType.
• Publish the ltVOResourcegt in IVO registry.

17
How to call an app-service using CEA?

• Look it up in the resource registry.
• Get the ltApplicationDefinitiongt from its
  ltVOResourcegt.
• Give the ltApplicationDefinitiongt to a CEA
  delegate gt define SOAP call.
• Add parameters from UI according to
  ltApplicationDefinitiongt.
• Bake in pre-heated SOAP engine _at_ HTTP 1.1.

18
CEA language example

• ltApplicationDefinitiongt
• ltParametersgt
• ltParameterDefinition
  name"DetectionImage" type"binary"gt
• ltUI_NamegtDetection
  Imagelt/UI_Namegt
• ltUI_DescriptiongtThe image
  that is used to detect sources.
• Basic position, shape and size information is
  derived from this imagelt/UI_Descriptiongt
• lt/ParameterDefinitiongt
• ltParameterDefinition
  name"PhotoImage" type"binary"gt
• ltUI_NamegtMeasurement
  ImageltUI_Namegt
• ltUI_DescriptiongtThe Image
  that is used to measure photometric
  parameterslt/UI_Descriptiongt
• lt/ParameterDefinitiongt
• ltParameterDefinition
  name"config_file" type"text"gt

19
CEA language schema (1)
20
CEA language schema (2)
21
CEA facilities so far
Other service
MySpace
22
Future evolution

• Add facilities to CEC
• Asynchronicity
• Security
• More parameter types
• New facility field-programmable CEC (PCEC)

23
PCEC (1) local code libraries
Client
Select exe
PCEC
Load exe
Code library
Data mine
24
PCEC (2) distributed code-library
VOSpace
Client
Code library
Upload exe
Select exe
PCEC
Download exe
Sandbox
Run exe
Data mine