Wayne Schroeder

1
IRODS Integrated Rule-Oriented Data-management
System
Wayne Schroeder Data Intensive Cyber Environments
Team (DICE) DICE Center, University of North
Carolina at Chapel Hill Institute for Neural
Computation (INC), University of California San
Diego irods.org
2
iRODS

• Research Problems Being Addressed
• General Data Management
• Mark Ellisman Distributed Collection Management
• Types of Data Being Used
• Wide variety
• Middle-ware in multiple research projects
• Tools That Can Be Shared
• iRODS, Open Source (BSD)

3
Science and Engineering Domains using the iRODS
Policy-based data management system Astrophysics
Auger supernova search Atmospheric
science NASA Langley Atmospheric Sciences
Center Biology Phylogenetics at CC
IN2P3 Climate NOAA National Climatic Data
Center Cognitive Science Temporal Dynamics of
Learning Center Computer Science GENI
experimental network Cosmic Ray AMS experiment
on the International Space Station Dark Matter
Physics Edelweiss II Earth Science NASA
Center for Climate Simulations Ecology CEED
Caveat Emptor Ecological Data Engineering CIBER
-U High Energy Physics BaBar / Stanford Linear
Accelerator Hydrology Institute for the
Environment, UNC-CH Hydroshare Genomics Broad
Institute, Wellcome Trust Sanger Institute,
NGS Medicine Sick Kids Hospital Neuroscience
International Neuroinformatics Coordinating
Facility Neutrino Physics T2K and dChooz
neutrino experiments Oceanography Ocean
Observatories Initiative Optical
Astronomy National Optical Astronomy
Observatory Particle Physics Indra
multi-detector collaboration at IN2P3 Plant
genetics the iPlant Collaborative Quantum
Chromodynamics IN2P3 Radio Astronomy Cyber
Square Kilometer Array, TREND, BAOradio Seismology
Southern California Earthquake Center Social
Science Odum, TerraPop
4
Not Just Large Projects

• John Helly(SDSC),
• Used SRB in multiple smaller projects
• Now using iRODS
• California Spatial Data Infrastructure (CSDI)
• Research ship Antarctic
• Laptops/PCs to HPC
• Storage Resource Broker (SDSC SRB)
• Predecesor to iRODS, 1995-2006
• IN2P3 used to 2013 (concurrent with iRODS)
• Nirvana Storage (General Atomics) commercial
  version

5
UCSD Projects using iRODS

• Temporal Dynamics of Learning Center (TDLC), DFC
• Andrea Chiba
• Ocean Observatories Initiative (OOI), DFC
• John Orcutt
• CineGrid
• California Spacial Data Infrastructure (CSDI)
• John Helly
• Biomedical Information Research Network (BIRN)

6
(No Transcript)
7
DICE Technologies Helping UCSD Projects

• The National Center for Microscopy and Imaging
  Research (NCMIR) is using DICE SRB and testing
  iRODS in the Cell Centered Database project.
• DICE iRODS helps computational seismologists from
  the Southern California Earthquake Center (SCEC)
  manage large-scale earthquake simulation data at
  SDSC and other TeraGrid sites.
• UCSD Libraries Digital Asset Management System
  (DAMS) using DICE technologies, including SRB.
• DICE iRODS helps Ocean Observatories Initiative
  (OOI) with Scripps and Calit2 manage large-scale
  diverse ocean data, including real-time streaming
  data.
• And others including CineGrid, TDLC, etc.

8
iRODS Team

• Data Intensive Cyber Environments, University of
  North Carolina, Chapel Hill, School of
  Information and Library Science (SILS)
• Reagan Moore (Professor), Arcot Rajasekar
  (Professor) (Raja), Hao Xu (Post-doc), Mike
  Conway, et al
• DICE-UCSD
• Wayne Schroeder, Sheau-yen Chen, Mike Wan
  (retired)
• Renaissance Computing Institute (RENCI), UNC
• Brand Fortner, Charles Schmitt, Leesa Brieger,
  Jason Coposky, Antoine de Torcy, et al

9
Mark van de Sanden, Dutch National HPC Centre, at
EUDAT conference
10
(No Transcript)
11
iRODS Unified Virtual Collection
11
12
A RENCI Data Grid
A complete data grid (zone) has one metadata
catalogue (iCAT)
NCSU
Duke
iRODS Server
iRODS Server
RENCI, Europa Center
ECU
UNC-A
UNC-CH
iRODS Server
Metadata Catalog (iCAT)
iRODS Server
iRODS Server
iRODS Server

• Client asks for data request goes to an iRODS
  server
• Server contacts the iCAT-enabled server (IES)
• Information (location, access rights, etc) is
• retrieved from the iCAT
• Server containing data is signaled to send data
  to
• authorized client

12
Data grids can be federated - the metadata
catalogues establish a trust relationship.
13
iRODS Distributed Data Management
14
DataNet Federation Consortium

• Management Reagan W. Moore (UNC-CH, PI)
• Mary Whitton (UNC-CH, Project Manager)
• Karl Gustafson (RENCI, Admin)
• Engineering William Regli (Drexel University,
  co-PI)
• Isaac Simons (Drexel University)
• Alexandru Nedelcu (Drexel University, REU)
• Hydrology Jonathan Goodall (USC, co-PI)
• Ken Galluppi (ASU)
• Mirza Billah (USC, GRA)
• Brian Miles (UNC-CH, GRA)
• Alan Hall (NCDC, collaborator)
• Oceanography John Orcutt (UCSD, co-PI)
• Wayne Schroeder (UCSD)

14
DFC April 2013 NSF Review --3.1--1
15
DFC Vision

• Enable collaborative research
• Sharing of data, information, and knowledge
• Build national data cyberinfrastructure
• Federation of existing data management systems
• Support reproducible data-driven research
• Encapsulate knowledge in shared workflows
• Enable student participation in research
• Policy-controlled access to live data

NEW
DFC April 2013 NSF Review --3.1--3
16
Data Driven Science and Engineering

• Collaboration Environments
• Oceanography Ocean Observatory Initiative
• Archiving climatic data records from real-time
  sensor data streams
• Engineering CIBER-U
• Engineering Digital Library Curating civil
  engineering data, materials data, archaeology
  data, student training materials
• Hydrology - EarthCube
• Automating hydrology research workflows (data
  retrieval, transformation, analysis)

Engineering Representation
DFC April 2013 NSF Review --3.1--4
17
iRODS Challenges

• Our Focus is on National and International User
  Sites
• Entering new phase
• NSF no longer funding general community support
  (SDCI)
• Integration and Interoperability
• Research
• As well as an applied and reliable tool
• Highly Configurable and Complex
• Requires someone to 'Bridge the Gap'
• Depend on User Sites to do that

18
iRODS Future

• DFC (2 years completed, 3 years phase 2)
• 1 Drexel, OOI, CUAHSI (Hydrologic)
• 2 iPlant, TDLC, Odum (Social Science)
• RENCI E-iRODS and iRODS 3.3.1 Merger
• 1H 2014
• IRODS 4.0
• RENCI/DICE iRODS Consortium
• Commercial Start-up
• Archive Analytics Paul Watry (Liverpool), John
  Burns, Wayne Schroeder, et al Alloy

19
iRODS
www.irods.org schroeder_at_diceresearch.org