iRODS Applications

1
iRODS Applications
Reagan W. Moore Mike Wan Arcot Rajasekar Wayne
Schroeder San Diego Supercomputer
Center moore,mwan, sekar, schroede_at_sdsc.edu htt
p//irods.sdsc.edu http//www.sdsc.edu/srb/
2
iRODS Applications

• Data grids - Share data
• Project level data sharing
• Digital libraries - Publish data
• Specify data context, provide standard services
• Persistent archive - Preserve data
• Build reference collections
• Real-time sensor systems - Federate data
• Manage real-time data distribution
• Workflow systems - Analyze data
• Integrate client- server-side workflows

3
Common Requirements

• iRODS software framework
• Production support mechanisms
• Basic clients
• Standard rule set
• Standard micro-services

4
iRODS Production Support

• iRODS releases every 4-6 months
• Wiki http//irods.sdsc.edu
• Chat - iROD-Chat_at_googlegroups.com
• CVS source code repository distributed with
  each release. Production CVS at SDSC
• Bugzilla - https//www.irods.org/bugzilla/
• Testing NMI testbed at University of Wisconsin,
  SDSC Tinderbox system
• Web site - http//diceresearch.org

5
iRODS Tutorials - 2008

• January 31, SDSC
• April 8 - ISGC, Taipei
• May 13 - China, National Academy of Science
• May 27-30 - UK eScience, Edinburgh
• June 5 - OGF23, Barcelona
• July 7-11 - SAA, SDSC
• August 4-8 - SAA, SDSC
• August 25 - SAA, San Francisco

6
Basic iRODS Clients

• C library calls
• iRODS rich web client
• https//rt.sdsc.edu8443/irods/index.php
• Unix shell commands
• iRODS/clients/icommands/bin
• FUSE user level file system
• iRODS/clients/fuse/bin/irodsFs fmount
• Jargon Java I/O class library
• iRODS/java/jargon
• Web browser and PHP client library
• http//irods.sdsc.edu

7
iRODS Web Client
audit1
8
Web Browser Interface
9
iCommands /irods/clients/icommands/bin

• iget
• iput
• ireg
• irepl
• itrim
• irsync
• ilsresc
• iphymv
• irmtrash
• ichksum
• iinit
• iexit

• iqdel
• iqmod
• iqstat
• iexecmd
• irule
• iuserinfo
• isysmeta
• imeta
• iquest
• imiscsvrinfo
• iadmin

• icd
• ichmod
• icp
• ils
• imkdir
• imv
• ipwd
• irm
• ienv
• ierror

10
Additional iRODS Clients

• Parrot user level file system
• Douglas Thain, Notre Dame University
• Fedora digital library
• Initial port - Bing Zhu
• Mapping of Fedora behaviors to iRODS
  micro-services - Mark Hedges, Kings College
  London
• DSpace digital library
• Initial port - MIT
• Web client (Python)
• https//www.irods.org/index.php/Web_Client

11
Python-based Web Client

• Basic interface for multi-browser support
• http//pho27.sdsc.edu/rodsclient/

12
Mapping Application Requirements to Rule Sets
iRODS - integrated Rule-Oriented Data System
13
Standard Rules

• Classes of rules
• Internal rules used to maintain consistency
  between operations and persistent state
  information
• Administrator controlled rules that are
  automatically invoked
• User executable rules

14
Rules

• Rule execution
• Atomic rules - executed on each operation invoked
  by a client
• Deferred rules - executed at a future time
• Periodic rules - executed to validate assessment
  criteria and enforce desired properties

15
iRODS Rule Syntax

• Event Condition Action-set Recovery-set
• Event - triggered by operation or queued rule
• Condition - composed by tests on any attributes
  in
• the persistent state information
• Action-set - composed from both micro-services
• and rules
• Recovery-set - used to ensure transaction
  semantics
• and consistent state information

16
iRODS Rule Sample
irule -F showcore.ir 5 core.acCreateUser
msiCreateUser msiRollback
acCreateDefaultCollections
msiRollback msiCommit
7 core.acCreateDefaultCollections
acCreateUserZoneCollections 8
core.acCreateUserZoneCollections
acCreateCollByAdmin(/rodsZoneProxy/home,otherUse
rName) acCreateCollByAdmin(/rodsZoneProxy/tras
h/home,otherUserName) 9
core.acCreateCollByAdmin(parColl,childColl)
msiCreateCollByAdmin(parColl,childCo
ll)
17
Standard Micro-service Classes

• Test micro-services
• System micro-services
• Workflow micro-services
• System micro-services
• User micro-services called by client
• iCAT micro-services
• User micro-services invoked by irule
• Image manipulation micro-services

18
Example Micro-Serviceshttp//irods.sdsc.edu/index
.php/List_of_Micro-Services

• Workflow Services
• nop, null - no action
• cut - not to retry any other
• applicable rules for this action
• succeed - succeed immediately
• fail - fail immediately - recovery
  and
• retries are possible
• msiGoodFailure - useful when you want to
  fail
• but no recovery initiated.
• msiNullAction - same as nop
• whileExec - while loop over result set
• forExec - for loop over result set

19
Example Micro-Serviceshttp//irods.sdsc.edu/index
.php/List_of_Micro-Services

• System Micro Services - Can only be called by the
  server process.
• msiSetDefaultResc - set the default resource
• msiSetNoDirectRescInp - sets a list of
  resources that cannot
• be used by a normal user directly.
• msiSetRescSortScheme - set the scheme for
  selecting the best
• resource to use
• msiSetMultiReplPerResc - sets the number of
  copies per resource
• to unlimited
• msiSetDataObjPreferredResc - if the data has
  multiple copies,
• specify the preferred copy to use
• msiSetDataObjAvoidResc - specify the copy to
  avoid
• msiSortDataObj - Sort the replica randomly
  when
• choosing which copy to use
• msiSetNumThreads - specify the parameters for
  determining
• the number of threads to use for data
  transfer.
• msiSysChksumDataObj - checksum a data object.
• msiSysReplDataObj - replicate a data object.

20
Example Applications

• Cognitive Science data grid
• Temporal Dynamics of Learning Center
• Human Subject Approval flags
• Preservation Environments
• Transcontinental Persistent Archive Prototype
• Trusted Repository Assessment Criteria

21
TDLC Administrative Database

• Manage IRB approval flags and track project
  information
• Share project information with partners
• Simplify production of annual progress reports

ADD EDIT
1
DATA GRID ACCESS
4
22
Example Workflow for A to access IRB/IACUC
restricted data of B
Administrative Database
B registers the IRB/IACUC approval in the TDLC
administrative database
B grants access permission to A
B gets IRB/IACUC approval for access by A to
the data
iRODS Data Grid
iRODS modifies the IRB/IACUC flag to change
the access permission
iRODS rule checks the TDLC database periodically
A can start accessing the data
A could be a person or a group of people.
data could be a collection, a single file, or
files belonging to a project.
23
Digital Preservation

• Preservation manages communication from the past
• What information do we need from the past to make
  assertions about preservation assessment criteria
  (authenticity, integrity, chain of custody)?
• RLG/NARA Trusted Repository Audit and
  Certification Criteria

24
RLG/NARA - TRAC Criteria

• Assessment categories
• Organizational infrastructure
• Digital Object Management
• Technologies, Technical Infrastructure and
  Security
• Example criteria
• B6.4 Repository has documented and implemented
  access policies (authorization rules,
  authentication requirements) consistent with
  deposit agreements for stored objects.

25
Mapping to a Set of Rules

• List staff who have archivist execution
  permission on collection
• List all persons with access permissions on
  collection
• Analyze audit trails to verify identity of all
  persons accessing the data, and compare their
  roles with desired access controls
• Generate report listing all persons who accessed
  or applied archival functions on the collection
• Compare report with the deposition agreement

26
Types of Information

• Persistent State Information
• Archivists - User name space
• Records - File name space
• Storage - Resource name space
• Policies - Rule name space
• Procedures - Micro-service name space
• Criteria - Status and audit

27
Automation

• Automate administrative tasks to minimize labor
  requirements
• Integrity checks - verify checksums periodically,
  replace bad files
• Chain of custody checks - parse audit trails to
  verify all operations have been performed by
  authorized archivist
• Authenticity checks - compare descriptive
  metadata with a list of required metadata

28
Applications in Progress

• Integration of Kepler workflow with iRODS
• Kepler access data stored in iRODS through Kepler
  actor
• iRODS invoke Kepler workflow through a
  micro-service
• Micro-services that invoke remote web services
• Manage structured information exchange between
  web services

29
iRODS Rules that Invoke Remote Web Services - NVO

• getObjPositionByName.ir
• Executes the micro-service msiObjByName
• Shows as output
• RA, Dec and Type
• getCutOutByPosition.ir
• Executes the micro-service msiSdssImgCutout_GetJp
  eg
• Stores the Image Cutout as a file in iRODS.
• Uses other iRODS system micro-services
• getCutOutByByObjName.ir
• Chains two web-services from two service
  providers
• Takes an Object Name, Cutout Parameters and
  stores an image cutout in an iRODS file

30
Applications in Progress

• Knowledge grid based on iRODS
• Incorporate Cheshire analysis utilities as iRODS
  micro-services
• Port Knowledge interface onto Fedora, which is
  ported on top of iRODS
• Image tools
• Integration of Imagemagick tool kit as iRODS
  micro-services
• http//irods.org/index.php/Properties_micro_servic
  es
• Integration of JHOVE type identification tools as
  iRODS micro-services

31

• Image Basics Micro-services
• --------------------------
• msiImageConvert( source, destination, format )
• Convert from a source file to a destination file
  using the
• named file format.
• Source and destination can be file paths or file
  descriptors.
• Format is a string.
• Use on data ingestion to convert to a
  standardized internal
• format (such as TIFF). Pre-convert and cache
  images into
• multiple formats for faster delivery. Convert
  as needed to
• an output format used for previewing content.
• Examples
• msiImageConvert( from.gif, to.jpg, jpg )
• msiImageConvert( SRCFD, to.jpg, jpg )
• msiImageConvert( SRCFD, DSTFD, jpg )
• msiImageConvert( from.gif, DSTFD, jpg )

32
For More Information

• Reagan W. Moore
• San Diego Supercomputer Center
• moore_at_sdsc.edu
• http//www.sdsc.edu/srb/
• http//irods.sdsc.edu/