Virtual Data Management for Grid Computing

1
Virtual Data Management for Grid Computing

• Michael Wilde
• Argonne National Laboratory
• wilde_at_mcs.anl.gov
• Gaurang Mehta, Karan Vahi
• Center for Grid Technologies
• USC Information Sciences Institute
• gmehta,vahi _at_isi.edu

2
(No Transcript)
3
Outline

• The concept of Virtual Data
• The Virtual Data System and Language
• Tools and approach for Running on the Grid
• Pegasus Grid Workflow Planning
• Virtual Data Applications on the Grid
• Research issues

4
The Virtual Data Concept

• Enhance scientific productivity through
• Discovery and application of datasets and
  programs at petabyte scale
• Enabling use of a worldwide data grid as a
  scientific workstation
• Virtual Data enables this approach by creating
  datasets from workflow recipes and recording
  their provenance.

5
Virtual Data Concept

• Motivated by next generation data-intensive
  applications
• Enormous quantities of data, petabyte-scale
• The need to discover, access, explore, and
  analyze diverse distributed data sources
• The need to organize, archive, schedule, and
  explain scientific workflows
• The need to share data products and the resources
  needed to produce and store them

6
GriPhyN The Grid Physics Network

• NSF-funded IT research project
• Supports the concept of Virtual Data, where data
  is materialized on demand
• Data can exist on some data resource and be
  directly accessible
• Data can exist only in a form of a recipe
• The GriPhyN Virtual Data System can seamlessly
  deliver the data to the user or application
  regardless of the form in which the data exists
• GriPhyN targets applications in high-energy
  physics, gravitational-wave physics and astronomy

7
Virtual Data System Capabilities

• Producing data from transformations with
  uniform, precise data interface descriptions
  enables
• Discovery finding and understanding datasets and
  transformations
• Workflow structured paradigm for organizing,
  locating, specifying, producing scientific
  datasets
• Forming new workflow
• Building new workflow from existing patterns
• Managing change
• Planning automated to make the Grid transparent
• Audit explanation and validation via provenance

8
Virtual Data ExampleGalaxy Cluster Search
DAG
Sloan Data
Galaxy cluster size distribution
Jim Annis, Steve Kent, Vijay Sehkri, Fermilab,
Michael Milligan, Yong Zhao,
University of Chicago
9
Virtual Data Application
High Energy Physics Data
Analysis
mass 200 decay WW stability 1 LowPt
20 HighPt 10000
Work and slide by Rick Cavanaugh and Dimitri
Bourilkov, University of Florida
10
Lifecycle of Virtual Data
I have some subject images, what analyses are
available? which can be applied to this format?
I need to document the devices and methods used
to measure the data, and keep track of the
analysis steps applied to the data.
Ive come across some interesting data, but I
need to understand the nature of the analyses
applied when it was constructed before I can
trust it for my purposes.
I want to apply an image registration program to
thousands of objects. If the results already
exist, Ill save weeks of computation.
11
A Virtual Data Grid
12
Virtual Data Scenario
Manage workflow
On-demand data generation
Update workflow following changes
Explain provenance, e.g. for file8
psearch t 10 i file3 file4 file5 o
file8summarize t 10 i file6 o file7reformat
f fz i file2 o file3 file4 file5 conv l esd
o aod i file 2 o file6simulate t 10 o file1
file2
13
VDL and Abstract Workflow
VDL descriptions
User request data file c
14
Example Workflow Reduction

• Original abstract workflow
• If b already exists (as determined by query to
  the RLS), the workflow can be reduced

15
Mapping from abstract to concrete

• Query RLS, MDS, and TC, schedule computation and
  data movement

16
Workflow management in GriPhyN

• Workflow Generation how do you describe the
  workflow (at various levels of abstraction)?
  (Virtual Data Language and catalog VDC))
• Workflow Mapping/Refinement how do you map an
  abstract workflow representation to an executable
  form? (Planners Pegasus)
• Workflow Execution how to you reliably execute
  the workflow? (Condors DAGMan)

17
Executable Workflow Construction

• VDL tools used to build an abstract workflow
  based on VDL descriptions
• Planners (e.g. Pegasus) take the abstract
  workflow and produce an executable workflow for
  the Grid or other environments
• Workflow executors (enactment engines) like
  Condor DAGMan execute the workflow

18
Terms

• Abstract Workflow (DAX)
• Expressed in terms of logical entities
• Specifies all logical files required to generate
  the desired data product from scratch
• Dependencies between the jobs
• Analogous to build style dag
• Concrete Workflow
• Expressed in terms of physical entities
• Specifies the location of the data and
  executables
• Analogous to a make style dag

19
Outline

• The concept of Virtual Data
• The Virtual Data System and Language
• Tools and Approach for Running on the Grid
• Pegasus Grid Workflow Planning
• Virtual Data Applications on the Grid
• Research issues

20
VDL Virtual Data LanguageDescribes Data
Transformations

• Transformation - TR
• Abstract template of program invocation
• Similar to "function definition"
• Derivation DV
• Function call to a Transformation
• Store past and future
• A record of how data products were generated
• A recipe of how data products can be generated
• Invocation
• Record of a Derivation execution

21
Example Transformation

• TR t1( out a2, in a1, none pa "500", none
  env "100000" )
• argument "-p "pa
• argument "-f "a1
• argument "-x y"
• argument stdout a2
• profile env.MAXMEM env

a1
t1
a2
22
Example Derivations

• DV d1-gtt1 (env"20000", pa"600",a2_at_outrun1.e
  xp15.T1932.summary,a1_at_inrun1.exp15.T1932.raw
  ,
• )
• DV d2-gtt1 (a1_at_inrun1.exp16.T1918.raw,a2_at_ou
  t.run1.exp16.T1918.summary
• )

23
Workflow from File Dependencies
file1

• TR tr1(in a1, out a2)
• argument stdin a1 
• argument stdout a2
• TR tr2(in a1, out a2)
• argument stdin a1
• argument stdout a2
• DV x1-gttr1(a1_at_infile1, a2_at_outfile2)
• DV x2-gttr2(a1_at_infile2, a2_at_outfile3)

x1
file2
x2
file3
24
Example Workflow

• Complex structure
• Fan-in
• Fan-out
• "left" and "right" can run in parallel
• Uses input file
• Register with RLS
• Complex file dependencies
• Glues workflow

preprocess
findrange
findrange
analyze
25
Workflow step "preprocess"

• TR preprocess turns f.a into f.b1 and f.b2
• TR preprocess( output b, input a ) argument
  "-a top"argument " i "inputaargument
  " o " outputb
• Makes use of the "list" feature of VDL
• Generates 0..N output files.
• Number file files depend on the caller.

26
Workflow step "findrange"

• Turns two inputs into one output
• TR findrange( output b, input a1, input a2,none
  name"findrange", none p"0.0" ) argument "-a
  "nameargument " i " a1 " "
  a2argument " o " bargument " p "
  p
• Uses the default argument feature

27
Can also use list parameters

• TR findrange( output b, input a,none
  name"findrange", none p"0.0" ) argument "-a
  "nameargument " i " " "aargument
  " o " bargument " p " p

28
Workflow step "analyze"

• Combines intermediary results
• TR analyze( output b, input a ) argument
  "-a bottom"argument " i " aargument "
  o " b

29
Complete VDL workflow

• Generate appropriate derivations
• DV top-gtpreprocess( b _at_out"f.b1", _at_
  out"f.b2" , a_at_in"f.a" )
• DV left-gtfindrange( b_at_out"f.c1",
  a2_at_in"f.b2", a1_at_in"f.b1", name"left",
  p"0.5" )
• DV right-gtfindrange( b_at_out"f.c2",
  a2_at_in"f.b2", a1_at_in"f.b1", name"right" )
  
• DV bottom-gtanalyze( b_at_out"f.d", a
  _at_in"f.c1", _at_in"f.c2" )

30
Compound Transformations

• Using compound TR
• Permits composition of complex TRs from basic
  ones
• Calls are independent
• unless linked through LFN
• A Call is effectively an anonymous derivation
• Late instantiation at workflow generation time
• Permits bundling of repetitive workflows
• Model Function calls nested within a function
  definition

31
Compound Transformations (cont)

• TR diamond bundles black-diamonds
• TR diamond( out fd, io fc1, io fc2, io fb1, io
  fb2, in fa, p1, p2 )
• call preprocess( afa, b outfb1,
  outfb2 )
• call findrange( a1infb1, a2infb2,
  name"LEFT", pp1, boutfc1 )
• call findrange( a1infb1, a2infb2,
  name"RIGHT", pp2, boutfc2 )
• call analyze( a infc1, infc2 ,
  bfd )

32
Compound Transformations (cont)

• Multiple DVs allow easy generator scripts
• DV d1-gtdiamond( fd_at_out"f.00005",
  fc1_at_io"f.00004", fc2_at_io"f.00003",
  fb1_at_io"f.00002", fb2_at_io"f.00001",
  fa_at_io"f.00000", p2"100", p1"0" )
• DV d2-gtdiamond( fd_at_out"f.0000B",
  fc1_at_io"f.0000A", fc2_at_io"f.00009",
  fb1_at_io"f.00008", fb2_at_io"f.00007",
  fa_at_io"f.00006", p2"141.42135623731", p1"0"
  )
• ...
• DV d70-gtdiamond( fd_at_out"f.001A3",
  fc1_at_io"f.001A2", fc2_at_io"f.001A1",
  fb1_at_io"f.001A0", fb2_at_io"f.0019F",
  fa_at_io"f.0019E", p2"800", p1"18" )

33
Functional MRI Analysis
34
fMRI Example AIR Tools
TR airalign_warp( in reg_img, in reg_hdr, in
sub_img, in sub_hdr, m, out
warp ) argument reg_img argument
sub_img argument warp
argument "-m " m argument "-q" TR
airreslice( in warp, sliced, out sliced_img,
out sliced_hdr ) argument warp
argument sliced
35
fMRI Example AIR Tools
TR airwarp_n_slice( in reg_img, in
reg_hdr, in sub_img, in sub_hdr, m "12",
io warp, sliced, out sliced_img, out sliced_hdr
) call airalign_warp( reg_imgreg_img,
reg_hdrreg_hdr,
sub_imgsub_img, sub_hdrsub_hdr,
mm,
warp outwarp ) call airreslice(
warpinwarp, slicedsliced,
sliced_imgsliced_img, sliced_hdrsliced_hdr
) TR airsoftmean( in sliced_img, in
sliced_hdr, arg1 "y", arg2 "null",
atlas, out atlas_img, out atlas_hdr )
argument atlas argument arg1 " "
arg2 argument sliced_img
36
fMRI Example AIR Tools
DV airi3472_3-gtairwarp_n_slice( reg_hdr
_at_in"3472-3_anonymized.hdr", reg_img
_at_in"3472-3_anonymized.img", sub_hdr
_at_in"3472-3_anonymized.hdr", sub_img
_at_in"3472-3_anonymized.img", warp
_at_io"3472-3_anonymized.warp", sliced
"3472-3_anonymized.sliced", sliced_hdr
_at_out"3472-3_anonymized.sliced.hdr",
sliced_img _at_out"3472-3_anonymized.sliced.img"
) DV airi3472_6-gtairwarp_n_slice(
reg_hdr _at_in"3472-3_anonymized.hdr",
reg_img _at_in"3472-3_anonymized.img",
sub_hdr _at_in"3472-6_anonymized.hdr",
sub_img _at_in"3472-6_anonymized.img", warp
_at_io"3472-6_anonymized.warp", sliced
"3472-6_anonymized.sliced", sliced_hdr
_at_out"3472-6_anonymized.sliced.hdr",
sliced_img _at_out"3472-6_anonymized.sliced.img"
)
37
fMRI Example AIR Tools
DV aira3472_3-gtairsoftmean( sliced_img
_at_in"3472-3_anonymized.sliced.img",
_at_in"3472-4_anonymized.sliced.img",
_at_in"3472-5_anonymized.sliced.img",
_at_in"3472-6_anonymized.sliced.img" ,
sliced_hdr _at_in"3472-3_anonymized.sl
iced.hdr", _at_in"3472-4_anonymized.sliced
.hdr", _at_in"3472-5_anonymized.sliced.hdr
", _at_in"3472-6_anonymized.sliced.hdr"
, atlas "atlas", atlas_img
_at_out"atlas.img", atlas_hdr
_at_out"atlas.hdr" )
38
Query Examples

• Which TRs can process a "subject" image?
• Q xsearchvdc -q tr_meta dataType subject_image
  input
• A fMRIDC.AIRalign_warp
• Which TRs can create an "ATLAS"?
• Q xsearchvdc -q tr_meta dataType atlas_image
  output
• A fMRIDC.AIRsoftmean
• Which TRs have output parameter "warp" and a
  parameter "options"
• Q xsearchvdc -q tr_para warp output options
• A fMRIDC.AIRalign_warp
• Which DVs call TR "slicer"?
• Q xsearchvdc -q tr_dv slicer
• A fMRIDC.FSLs3472_3_x-gtfMRIDC.FSLslicer
• fMRIDC.FSLs3472_3_y-gtfMRIDC.FSLslicer
  fMRIDC.FSLs3472_3_z-gtfMRIDC.FSLslicer

39
Query Examples

• List anonymized subject-images for young
  subjects. This query searches for files based on
  their metadata.
• Q xsearchvdc -q lfn_meta
• dataType subject_image
• privacy anonymized
• subjectType young
• A 3472-4_anonymized.img
• For a specific patient image, 3472-3, show all
  DVs and files that were derived from this image,
  directly or indirectly.
• Q xsearchvdc -q lfn_tree
• 3472-3_anonymized.img
• A 3472-3_anonymized.img
• 3472-3_anonymized.sliced.hdr
• atlas.hdr
• atlas.img
• atlas_z.jpg
• 3472-3_anonymized.sliced.img

40
Virtual Provenancelist of derivations and files
ltjob id"ID000001" namespace"Quarknet.HEPSRCH"
name"ECalEnergySum" level"5"
dv-namespace"Quarknet.HEPSRCH"
dv-name"run1aesum"gt ltargumentgtltfilename
file"run1a.event"/gt ltfilename file"run1a.esm"/gtlt
/argumentgt ltuses file"run1a.esm" link"output"
dontRegister"false" dontTransfer"false"/gt
ltuses file"run1a.event" link"input"
dontRegister"false dontTransfer"false"/gt lt/jobgt
... ltjob id"ID000014" namespace"Quarknet.HEPSRC
H" name"ReconTotalEnergy" level"3"
ltargumentgtltfilename file"run1a.mis"/gt ltfilename
file"run1a.ecal"/gt ltuses file"run1a.muon"
link"input" dontRegister"false"
dontTransfer"false"/gt ltuses file"run1a.total"
link"output" dontRegister"false"
dontTransfer"false"/gt ltuses file"run1a.ecal"
link"input" dontRegister"false"
dontTransfer"false"/gt ltuses file"run1a.hcal"
link"input" dontRegister"false"
dontTransfer"false"/gt ltuses file"run1a.mis"
link"input" dontRegister"false"
dontTransfer"false"/gt lt/jobgt lt!--list of all
files used --gt ltfilename file"ecal.pct"
link"inout"/gt ltfilename file"electron10GeV.avg"
link"inout"/gt ltfilename file"electron10GeV.sum
" link"inout"/gt ltfilename file"hcal.pct"
link"inout"/gt ... (excerpted for display)
41
Virtual Provenance in XMLcontrol flow graph
ltchild ref"ID000003"gt ltparent
ref"ID000002"/gt lt/childgt ltchild
ref"ID000004"gt ltparent ref"ID000003"/gt
lt/childgt ltchild ref"ID000005"gt ltparent
ref"ID000004"/gt
ltparent ref"ID000001"/gt... ltchild
ref"ID000009"gt ltparent ref"ID000008"/gt
lt/childgt ltchild ref"ID000010"gt ltparent
ref"ID000009"/gt
ltparent ref"ID000006"/gt... ltchild
ref"ID000012"gt ltparent ref"ID000011"/gt
lt/childgt ltchild ref"ID000013"gt ltparent
ref"ID000011"/gt lt/childgt ltchild
ref"ID000014"gt ltparent ref"ID000010"/gt
ltparent
ref"ID000012"/gt...
ltparent ref"ID000013"/gt...lt/childgt (excerpted
for display)
42
Invocation Provenance
Completion status and resource usage
Attributes of executable transformation
Attributes of input and output files
43
Future Provenance Workflowfor Web Services
44
(No Transcript)
45
Outline

• The concept of Virtual Data
• The Virtual Data System and Language
• Tools and Issues for Running on the Grid
• Pegasus Grid Workflow Planning
• Virtual Data Applications on the Grid
• Research issues

46
Outline

• Introduction and the GriPhyN project
• Chimera
• Overview of Grid concepts and tools
• Pegasus
• Applications using Chimera and Pegasus
• Research issues
• Exercises

47
Motivation for Grids How do we solve problems?

• Communities committed to common goals
• Virtual organizations
• Teams with heterogeneous members capabilities
• Distributed geographically and politically
• No location/organization possesses all required
  skills and resources
• Adapt as a function of the situation
• Adjust membership, reallocate responsibilities,
  renegotiate resources

48
The Grid Vision

• Resource sharing coordinated problem solving
  in dynamic, multi-institutional virtual
  organizations
• On-demand, ubiquitous access to computing, data,
  and services
• New capabilities constructed dynamically and
  transparently from distributed services

49
The Grid

• Emerging computational, networking, and storage
  infrastructure
• Pervasive, uniform, and reliable access to remote
  data, computational, sensor, and human resources
• Enable new approaches to applications and problem
  solving
• Remote resources the rule, not the exception
• Challenges
• Heterogeneous components
• Component failures common
• Different administrative domains
• Local policies for security and resource usage

50
Data Grids for High Energy Physics
Slide courtesy Harvey Newman, CalTech
www.griphyn.org www.ppdg.net
www.eu-datagrid.org
51
Grid Applications

• Increasing in the level of complexity
• Use of individual application components
• Reuse of individual intermediate data products
• Description of Data Products using Metadata
  Attributes
• Execution environment is complex and very dynamic
• Resources come and go
• Data is replicated
• Components can be found at various locations or
  staged in on demand
• Separation between
• the application description
• the actual execution description

52
Grid3 The Laboratory
Supported by the National Science Foundation and
the Department of Energy.
53
Globus Monitoring and Discovery Service (MDS)
The MDS architecture is a flexible hierarchy.
There can be several levels of GIISes any GRIS
can register with any GIIS and any GIIS can
register with another, making this approach
modular and extensible.
54
GridFTP

• Data-intensive grid applications transfer and
  replicate large data sets (terabytes, petabytes)
• GridFTP Features
• Third party (client mediated) transfer
• Parallel transfers
• Striped transfers
• TCP buffer optimizations
• Grid security
• Important feature is separation of control and
  data channel

55
A Replica Location Service

• A Replica Location Service (RLS) is a distributed
  registry service that records the locations of
  data copies and allows discovery of replicas
• Maintains mappings between logical identifiers
  and target names
• Physical targets Map to exact locations of
  replicated data
• Logical targets Map to another layer of logical
  names, allowing storage systems to move data
  without informing the RLS
• RLS was designed and implemented in a
  collaboration between the Globus project and the
  DataGrid project

56

• LRCs contain consistent information about
  logical-to-target mappings on a site
• RLIs nodes aggregate information about LRCs
• Soft state updates from LRCs to RLIs relaxed
  consistency of index information, used to rebuild
  index after failures
• Arbitrary levels of RLI hierarchy

57
Condors DAGMan

• Developed at UW Madison (Livny)
• Executes a concrete workflow
• Makes sure the dependencies are followed
• Executes the jobs specified in the workflow
• Execution
• Data movement
• Catalog updates
• Provides a rescue DAG in case of failure

58
Outline

• The concept of Virtual Data
• The Virtual Data System and Language
• Tools and Approach for Running on the Grid
• Pegasus Grid Workflow Planning
• Virtual Data Applications on the Grid
• Research issues

59
Pegasus Outline

• Introduction
• Pegasus and How it works
• Pegasus Components and Internals
• Deferred Planning
• Pegasus Portal

60
Introduction

• Scientific applications in various domains are
  compute and data intensive.
• Not feasible to run on a single resource (limits
  scalability).
• Scientific application can be represented as a
  workflow.
• Use of individual application components.
• Reuse data if possible (Virtual Data).

61
(No Transcript)
62
Generating an Abstract Workflow

• Available Information
• Specification of component capabilities
• Ability to generate the desired data products
• Select and configure application components to
  form an abstract workflow
• assign input files that exist or that can be
  generated by other application components.
• specify the order in which the components must be
  executed
• components and files are referred to by their
  logical names
• Logical transformation name
• Logical file name
• Both transformations and data can be replicated

63
Generating a Concrete Workflow

• Information
• location of files and component Instances
• State of the Grid resources
• Select specific
• Resources
• Files
• Add jobs required to form a concrete workflow
  that can be executed in the Grid environment
• Each component in the abstract workflow is turned
  into an executable job

64
Concrete Workflow Generator
Domain Knowledge
Resource Information
Location Information
Plan submitted to the grid
Concrete Workflow Generator
Abstract Workflow
65
Pegasus

• Pegasus - Planning for Execution on Grids
• Planning framework
• Take as input an abstract workflow
• Abstract Dag in XML (DAX)
• Generates concrete workflow.
• Submits the workflow to a workflow executor (e.g.
  Condor Dagman).

66
Pegasus Outline

• Introduction
• Pegasus and How it works
• Pegasus Components and Internals
• Deferred Planning
• Pegasus Portal

67
(No Transcript)
68
Abstract Workflow Reduction
69
Optimizing from the point of view of Virtual Data
Job c
Job a
Job b
Job f
Job e
Job d
Job g
Job h
Job i

• Jobs d, e, f have output files that have been
  found in the Replica Location Service.
• Additional jobs are deleted.
• All jobs (a, b, c, d, e, f) are removed from the
  DAG.

70
Planner picks execution and replica
locations Plans for staging data in
Job c
adding transfer nodes for the input files for the
root nodes
Job a
Job b
Job f
Job e
Job d
Job g
Job h
Job i
71
Staging data out and registering new derived
products
Job c
Job a
Job b
Job f
Job e
Job d
Job g
Job h
Staging and registering for each job that
materializes data (g, h, i ).
Job i
KEY The original node Input transfer
node Registration node Output transfer
node Node deleted by Reduction algorithm
72
(No Transcript)
73
Pegasus Outline

• Introduction
• Pegasus and How it works
• Pegasus Components and Internals
• Deferred Planning
• Pegasus Portal

74
Pegasus Components
75
Replica Location Service

• Pegasus uses the RLS to find input data

RLI
LRC
LRC
LRC

• Pegasus uses the RLS to register new data
  products

76
Pegasus Components
77
Transformation Catalog

• Transformation Catalog keeps information about
  domain transformations (executables).
• It keeps information like the logical name of the
  transformation, the resource on the which the
  transformation is available and the URL for the
  physical transformation.
• The new transformation catalog also allows to
  define different types of transformation,
  Architecture, OS, OS-version and glibc version of
  the transformation.
• Profiles which allow a user to attach Environment
  variables, hints, scheduler related information,
  job-manager configurations

78
Transformation Catalog

• A set of standard api's are available to perform
  various operations on the transformation Catalog.
• Various implementations are available for the
  transformation catalog like
• Database
• Useful for large Catalogs and supports fast adds,
  deletes and queries.
• Standard database security used.
• File Format
• Useful for small number of transformations and
  easy editing.
• isi vdsfft1.0 /opt/bin/fft INSTALLED
  INTEL32LINUX ENVVDS_HOME/opt/vds
• Users can provide their own implementation.

79
Pegasus Components
80
Resource Information Catalog(Pool Config)

• Pool Config is an XML file which contains
  information about various pools/sites on which
  DAGs may execute.
• Some of the information contained in the Pool
  Config file is
• Specifies the various job-managers that are
  available on the pool for the different types of
  schedulers.
• Specifies the GridFtp storage servers associated
  with each pool.
• Specifies the Local Replica Catalogs where data
  residing in the pool has to be cataloged.
• Contains profiles like environment hints which
  are common site-wide.
• Contains the working and storage directories to
  be used on the pool.

81
Resource Information CatalogPool Config

• Two Ways to construct the Pool Config File.
• Monitoring and Discovery Service
• Local Pool Config File (Text Based)
• Client tool to generate Pool Config File
• The tool gen-poolconfig is used to query the MDS
  and/or the local pool config file/s to generate
  the XML Pool Config file.

82
Monitoring and Discovery Service

• MDS provides up-to-date Grid state information
• Total and idle job queues length on a pool of
  resources (condor)
• Total and available memory on the pool
• Disk space on the pools
• Number of jobs running on a job manager
• Can be used for resource discovery and selection
• Developing various task to resource mapping
  heuristics
• Can be used to publish information necessary for
  replica selection
• Publish gridftp transfer statistics
• Developing replica selection components

83
Pegasus Components
84
Site Selector

• Determines job to site mappings.
• Selector Api provides easy pluggable
  implementations.
• Implementations provided
• Random
• RoundRobin
• Group
• NonJavaCallout
• Research on advance algorithms for site selector
  in progress

85
Pegasus Components
86
Transfer Tools / Workflow Executors

• Various transfer tools are used for staging data
  between grid sites.
• Define strategies to reliably and efficiently
  transfer data
• Current implementations
• Globus url copy
• Transfer
• Transfer2 (T2)
• Stork
• Workflow executors take the concrete workflow and
  act as a metascheduler for the Grid.
• Current supported workflow executors
• Condor Dagman
• Gridlab GRMS

87
System Configuration - Properties

• Properties file define and modify the behavior of
  Pegasus.
• Properties set in the VDS_HOME/etc/properties
  can be overridden by defining them either in
  HOME/.chimerarc or by giving them on the command
  line of any executable.
• eg. Gendax Dvds.homepath to vds home
• Some examples follow but for more details please
  read the sample.properties file in VDS_HOME/etc
  directory.
• Basic Required Properties
• vds.home This is auto set by the clients from
  the environment variable VDS_HOME
• vds.properties Path to the default properties
  file
• Default vds.home/etc/properties

88
Pegasus Clients in the Virtual Data System

• gencdag
• Converts DAX to concrete Dag
• genpoolconfig
• Generates a pool config file from MDS
• tc-client
• Queries, adds, deletes transformations from the
  Transformation Catalog
• rls-client, rls-query-client
• Queries, adds, deletes entry from the RLS
• partiondax
• Partitions the abstract Dax into smaller daxs.

89
(No Transcript)
90
Concrete Planner Gencdag

• The Concrete planner takes the DAX produced by
  Chimera and converts into a set of condor dag and
  submit files.
• You can specify more then one execution pools.
  Execution will take place on the pools on which
  the executable exists. If the executable exists
  on more then one pool then the pool on which the
  executable will run is selected depending on the
  site selector used.
• Output pool is the pool where you want all the
  output products to be transferred to. If not
  specified the materialized data stays on the
  execution pool
• Authentication removes sites which are not
  accessible.

91
Pegasus Outline

• Introduction
• Pegasus and How it works
• Pegasus Components and Internals
• Deferred Planning
• Pegasus Portal

92
Abstract To Concrete Steps
93
Original Pegasus configuration
Simple scheduling random or round robin using
well-defined scheduling interfaces.
94
Deferred Planning through Partitioning
A variety of partitioning algorithms can be
implemented
95
Mega DAG is created by Pegasus and then submitted
to DAGMan
96
Mega DAG Pegasus
97
Re-planning capabilities
98
Complex Replanning for Free (almost)
99
Planning Scheduling Granularity

• Partitioning
• Allows to set the granularity of planning ahead
• Node aggregation
• Allows to combine nodes in the workflow and
  schedule them as one unit (minimizes the
  scheduling overheads)
• May reduce the overheads of making scheduling and
  planning decisions
• Related but separate concepts
• Small jobs
• High-level of node aggregation
• Large partitions
• Very dynamic system
• Small partitions

100
Partitioned Workflow Processing

• Create workflow partitions
• partition the abstract workflow into smaller
  workflows.
• create the xml partition graph that lists out the
  dependencies between partitions.
• Create the MegaDAG (creates the dagman submit
  files)
• transform the xml partition graph to its
  corresponding condor representation.
• Submit the MegaDAG
• Each job invokes Pegasus on a partition and then
  submits the plan generated back to condor.

101
Future work for Pegasus

• Staging in executables on demand
• Expanding the scheduling plug-ins
• Investigating various partitioning approaches
• Investigating reliability across partitions

102
Pegasus Outline

• Introduction
• Pegasus and How it works
• Pegasus Components and Internals
• Deferred Planning
• Pegasus Portal

103
Portal

• Provides proxy based login
• Authentication to the sites
• Configuration for files
• Metadata based workflow creation
• Workflow Submission and Monitoring
• User Notification

104
(No Transcript)
105
Portal Demonstration
106
Outline

• The concept of Virtual Data
• The Virtual Data System and Language
• Tools and Issues for Running on the Grid
• Pegasus Grid Workflow Planning
• Virtual Data Applications on the Grid
• Research issues

107
Types of Applications

• Gravitational Wave Physics
• High Energy Physics
• Astronomy
• Earthquake Science
• Computational Biology

108
LIGO Scientific Collaboration

• Continuous gravitational waves are expected to be
  produced by a variety of celestial objects
• Only a small fraction of potential sources are
  known
• Need to perform blind searches, scanning the
  regions of the sky where we have no a priori
  information of the presence of a source
• Wide area, wide frequency searches
• Search is performed for potential sources of
  continuous periodic waves near the Galactic
  Center and the galactic core
• Search for binary inspirals collapsing into black
  holes.
• The search is very compute and data intensive

109
Additional resources used Grid3 iVDGL resources
110
LIGO Acknowledgements

• Patrick Brady, Scott Koranda, Duncan Brown,
  Stephen Fairhurst University of Wisconsin
  Milwaukee, USA
• Stuart Anderson, Kent Blackburn, Albert
  Lazzarini, Hari Pulapaka, Teviet Creighton
  Caltech, USA
• Gabriela Gonzalez, Louisiana State University
• Many Others involved in the Testbed
• www.ligo.caltech.edu
• www.lsc-group.phys.uwm.edu/lscdatagrid/
• LIGO Laboratory operates under NSF cooperative
  agreement PHY-0107417

111
Montage

• Montage (NASA and NVO)
• Deliver science-grade custom mosaics on demand
• Produce mosaics from a wide range of data sources
  (possibly in different spectra)
• User-specified parameters of projection,
  coordinates, size, rotation and spatial sampling.
  

Mosaic created by Pegasus based Montage from a
run of the M101 galaxy images on the Teragrid.
112
Small Montage Workflow
1200 nodes
113
Montage Acknowledgments

• Bruce Berriman, John Good, Anastasia Laity,
  Caltech/IPAC
• Joseph C. Jacob, Daniel S. Katz, JPL
• http//montage.ipac. caltech.edu/
• Testbed for Montage Condor pools at USC/ISI, UW
  Madison, and Teragrid resources at NCSA, Caltech,
  and SDSC.
• Montage is funded by the National Aeronautics
  and Space Administration's Earth Science
  Technology Office, Computational Technologies
  Project, under Cooperative Agreement Number
  NCC5-626 between NASA and the California
  Institute of Technology.

114
Other ApplicationsSouthern California Earthquake
Center

• Southern California Earthquake Center (SCEC), in
  collaboration with the USC Information Sciences
  Institute, San Diego Supercomputer Center, the
  Incorporated Research Institutions for
  Seismology, and the U.S. Geological Survey, is
  developing the Southern California Earthquake
  Center Community Modeling Environment (SCEC/CME).
• Create fully three-dimensional (3D) simulations
  of fault-system dynamics.
• Physics-based simulations can potentially provide
  enormous practical benefits for assessing and
  mitigating earthquake risks through Seismic
  Hazard Analysis (SHA).
• The SCEC/CME system is an integrated geophysical
  simulation modeling framework that automates the
  process of selecting, configuring, and executing
  models of earthquake systems.

Acknowledgments Philip Maechling and Vipin
Gupta University Of Southern California
115
Astronomy

• Galaxy Morphology (National Virtual Observatory)
• Investigates the dynamical state of galaxy
  clusters
• Explores galaxy evolution inside the context of
  large-scale structure.
• Uses galaxy morphologies as a probe of the star
  formation and stellar distribution history of the
  galaxies inside the clusters.
• Data intensive computations involving hundreds of
  galaxies in a cluster

The x-ray emission is shown in blue, and the
optical mission is in red. The colored dots are
located at the positions of the galaxies within
the cluster the dot color represents the value
of the asymmetry index. Blue dots represent the
most asymmetric galaxies and are scattered
throughout the image, while orange are the most
symmetric, indicative of elliptical galaxies,
are concentrated more toward the center.
People involved Gurmeet Singh, Mei-Hui Su, many
others
116
Biology Applications

• Tomography (NIH-funded project)
• Derivation of 3D structure from a series of 2D
  electron microscopic projection images,
• Reconstruction and detailed structural analysis
• complex structures like synapses
• large structures like dendritic spines.
• Acquisition and generation of huge amounts of
  data
• Large amount of state-of-the-art image processing
  required to segment structures from extraneous
  background.

Dendrite structure to be rendered by Tomography

• Work performed by Mei-Hui Su with Mark Ellisman,
  Steve Peltier, Abel Lin, Thomas Molina (SDSC)

117
BLAST set of sequence comparison algorithms that
are used to search sequence databases for optimal
local alignments to a query

• 2 major runs were performed using Chimera and
  Pegasus
• 60 genomes (4,000 sequences each),
• In 24 hours processed Genomes selected from
  DOE-sponsored sequencing projects
• 67 CPU-days of processing time delivered
• 10,000 Grid jobs
• gt200,000 BLAST executions
• 50 GB of data generated
• 2) 450 genomes processed
• Speedup of 5-20 times were achieved because the
  compute nodes we used efficiently by keeping the
  submission of the jobs to the compute cluster
  constant.

Lead by Veronika Nefedova (ANL) as part of the
Paci Data Quest Expedition program
118
Virtual Data ExampleGalaxy Cluster Search
DAG
Sloan Data
Galaxy cluster size distribution
Jim Annis, Steve Kent, Vijay Sehkri, Fermilab,
Michael Milligan, Yong Zhao,
University of Chicago
119
Cluster SearchWorkflow Graphand Execution Trace
Workflow jobs vs time
120
Capone Grid Interactions
ATLAS Slides courtesy of Marco Mambelli, U
Chicago ATLAS Team
121
ATLAS Capone Production Executor

• Reception
• Job received from work distributor
• Translation
• Un-marshalling, ATLAS transformation
• DAX generation
• VDL tools generate abstract DAG
• Input file retrieval from RLS catalog
• Check RLS for input LFNs (retrieval of GUID, PFN)
• Scheduling CE and SE are chosen
• Concrete DAG generation and submission
• Pegasus creates Condor submit files
• DAGMan invoked to manage remote steps

122
A job in Capone (2, execution)

• Remote job running / status checking
• Stage-in of input files, create POOL FileCatalog
• Athena (ATLAS code) execution
• Remote Execution Check
• Verification of output files and exit codes
• Recovery of metadata (GUID, MD5sum, exe
  attributes)
• Stage Out transfer from CE site to destination
  SE
• Output registration
• Registration of the output LFN/PFN and metadata
  in RLS
• Finish
• Job completed successfully, communicates to
  Windmill that jobs is ready for validation
• Job status is sent to Windmill during all the
  execution
• Windmill/DQ validate register output in ProdDB

123
Ramp up ATLAS DC2
124
Outline

• The concept of Virtual Data
• The Virtual Data System and Language
• Tools and approach for Running on the Grid
• Pegasus Grid Workflow Planning
• Virtual Data Applications on the Grid
• Research issues

125
Research directions inVirtual Data Grid
Architecture
126
Research issues

• Focus on data intensive science
• Planning is necessary
• Reaction to the environment is a must (things go
  wrong, resources come up)
• Iterative Workflow Execution
• Workflow Planner
• Workload Manager
• Planning decision points
• Workflow Delegation Time (eager)
• Activity Scheduling Time (deferred)
• Resource Availability Time (just in time)
• Decision specification level
• Reacting to the changing
• environment and recovering from failures
• How does the communication takes place?
  Callbacks, workflow annotations etc

Abstract workflow
Planner
Concrete workflow
info
Manager
Tasks
info
Resource Manager
Grid
127
Provenance Hyperlinks
128
Goals for a Dataset Model
ltFORM ltTitlegt /FORMgt
File
Set of files
Object closure
XML Element
Relational query or spreadsheet range
Set of files with relational index
New user-defined dataset type
Speculative model described in CIDR 2003 paper by
Foster, Voeckler, Wilde and Zhao
129
Acknowledgements
GriPhyN, iVDGL, and QuarkNet(in part) are
supported by the National Science Foundation
The Globus Alliance, PPDG, and QuarkNet are
supported in part by the US Department of Energy,
Office of Science by the NASA Information Power
Grid program and by IBM
130
Acknowledgements

• Argonne and The University of Chicago Ian
  Foster, Jens Voeckler, Yong Zhao, Jin Soon Chang,
  Luiz Meyer (UFRJ)
• www.griphyn.org/vds
• The USC Information Sciences InstituteEwa
  Deelman, Carl Kesselman, Gurmeet Singh, Mei-Hui
  Su
• http//pegasus.isi.edu
• Special thanks to Jed Dobson, fMRI Data Center,
  Dartmouth College

131
For further information

• Chimera and Pegasus
• www.griphyn.org/chimera
• http//pegasus.isi.edu
• Workflow Management research group in GGF
• www.isi.edu/deelman/wfm-rg
• Demos on the SC floor
• Argonne National Laboratory andUniversity of
  Southern California Booths.