CCSDS XML WORKSHOP ESA Activities

1
CCSDS XML WORKSHOPESA Activities
Nestor Peccia
2
XML Activities at ESOC

• Overview
• An ESA Framework The EVA Initiative
• XML Packaging Study at ESOC
• CEOS ICF at ESRIN
• The TAMINO case study
• Ingestion Methodology Proposal
• ESRIN - DEBAT Study

3
ESA Virtual Archive

• WG created at ESA level involving representatives
  of the main directorates
• Composition
• TOS-G N. Peccia Chairman
• APP-A G. M. Pinna Member
• TOS-M H P de Koning Member
• SCI-S C. Arviset Member
• Scope
• This cross-functional, cross-Directorate forum
  shall
• provide a forum for exchange of information
  between the different archive developers/operators
  within ESA and
• identify possible common building blocks for ESA
  archives.

4
EVA WG

• Activities
• Produce a survey of internal / external to ESA
  existing Archival Systems, new initiatives and
  technology trends.
• Identify common services, infrastructure software
  (middleware) and tools that could be used as
  building blocks by any ESA Archive Project.
• Identify essential enabling technologies, such as
  information exchange protocols, metadata
  standards and data preservation techniques.
• Identify those items which could usefully be
  addressed within the framework of an EVA
  initiative (e.g. definition of common
  requirements, ESA network bandwidth, Internet 2
  technology, shared computing resources in a GRID
  architecture, etc)
• Present an initial report to the audience of the
  1st EVA meeting by beginning May 2001.

5
Archive Survey

• The following archives were selected for the
  survey
• ISO Data Archive
• XMM SOC AMS
• ESRIN AMSESOC Generic DDS
• ESRIN MUIS
• CNES Plasma Physics Archive (SIPAD)
• JPL PSD
• Envisat PDS
• ERS HDR archive in RAL
• Last three archives were dropped due to lack of
  data and / or time.
• It was decided in the last general meeting to
  complete the survey with the following Archives
• HEASARC Archive
• Envisat PDS Archive
• An Archive based on an ODBMS
• ESO ST Archive

6
Archive Survey First conclusions

• Good mapping of ESA Archives to OAIS reference
  Model
• Main differences are as follows
• No clear separation between content information
  and Preservation Descriptive Information
• Prime and backup chains are not physically
  separated to guarantee an smoothly disaster
  recovery
• No file error checking when archiving (e.g.
  checksum)
• Administrative functions covered either via ICDs,
  documents produced during SR / AD Phases or done
  by S/W support
• Preservation Planning functions done by Archive
  Scientist and / or Archive Administrator
• Comparison against other archival aspects
• Main differences are as follows
• Variety of Data Interchange Standards, DBs, H/W,
  S/W and S/W languages used
• Limited usage of Data Dictionaries
• Metadata delivery / visualisation method
  different on each archive
• Limited or no interoperability

7
Archive Survey - General Comments

• Interoperability
• Requirements for exchange between ESA Archives
  have to be clarified. Interoperability is only
  needed for archives of data of the same
  discipline (therefore interagency) and not for
  archives of data of different disciplines in the
  same agency. EO missions have more
  common (but exception is the exchange between EO
  and planetary missions, where similar user
  services are envisaged problems with DLR or other
  national Agencies than with D/SCI for Mars
  express and Envisat -mutatis mutandis-). The same
  applies to D/SCI and its relations with the
  scientific community.
• Catalogue Interoperability should be
  distinguished from data interoperability.

8
EVA High Level Concept
What EVA really is not?
An standard to be imposed for developing ESA
Archives
Ø

9
EVA High Level Concept

• The major capabilities of EVA that need to be
  established in order to enable its goals include
  the ability to
• Build tools and services upon open standards
  (e.g. Java, XML, DEDSL, OAIS)
• Define a metadata language for catalogues and
  data sets
• Provide a framework for incorporating new data.
• Link with existing and future digital libraries.
• Similar initiatives are being exploited on the
  following projects
• The US NVO (National Virtual observatory)
• The CERN Data Grid, on which ESA is participating
  via ESRIN on Earth Observation Applications.
• JPL OODT
• GSFC SML
• SPACE GRID
• The European Astronomical Virtual Observatory

10
ESOC XML PACKAGING STUDY

• ESOC Study on XML Packaging ( 01.06.01 - 28.02.02
  )
• ESRIN and ESTEC-SSD involved by including their
  Archives (IDA, MUISlt AMS) in the proposed survey.
• Study Phases
• Consolidation (TN, URD, SRD) (01.06.01 -
  31.10.01)
• Comparison of Archives
• XML survey
• Specification of Mechanism
• selection of Archive
• Prototype development (01.11.01 - 31.01.02)
• Two totally different Archives will be selected
  to demonstrate inter-operability
• Installation presentation at ESOC (01.02.02 -
  28.02.02)

11
ESOC XML PACKAGING STUDY

• XML Packaging Issues
• XML SFDUs or XML plain
• Binary data (raw TM / TC packets Auxiliary
  files) encoded in XML or transferred
  independently
• Metadata described using XML DTD or Schema
• Translators needed for archive interoperability
• DDL used for syntax and semantic data description
  (EAST based, XML based?)

12
ESOC ARCHIVES - A future vision
PDI
Packaging Info
Descriptive Info
CAOS
XML ?
Global DED
DEDs (XML based)
XML ?
Syntax / Semantic data desciption via DDL
DIP / SIP
DIP / SIP
User, PIs SOC, SDC
DIP / SIP
. . .
Raw Data Binary Files
. . .
Aux Data Binary Files
Via ftp, CD, DVD, http, XML ? XML SFDU ?
13
CEOS ICF (InterChange Format)

• Main Concepts General Requirements
• Make exchange of archive data among facilities
  easier
• Decrease the number of archive format translators
• BaseBand compliant
• Developed by the CEOS WGISS DS ATT
• CEOS Committee on Earth Observation Satellites
• WGISS Working Group on Information Systems and
  Services
• DS Data Subgroup
• ATT Archive Task Team
• ESA in charge of ICF generation
• Demo translation SW (FRED ? ICF ? WILMA)

14
CEOS ICF

• Main Concept Properties
• METADATA FILE

• Interchange
• BaseBand data concept
• Multi-missions and multi-sensors
• Extendibility
• Hardware/software and OS independence

• CEOS ICF metadata file format is XML

• XML namespace
• Http//wgiss.Ceos.Org/ceos-icf/ceos-icf-v1.0-09042
  001 (TBC)

• Validation (XML schema)
• Http//wgiss.Ceos.Org/ceos-icf/ceos-icf-v1.0-09042
  001.Xsd (TBC)

15
ESOC DDS with Tamino
Spacecraft
MCS
Server
XML SML
MCS Format
DDS User
SOAP-XML Internet Intranet
RDM Production System
SOAP-XML Internet Intranet
Server
XSLT
XSLT
SOAP / XML can be used as HTTP communincation
protocol for Internet and Intranet and offers a
flexible distribution of systems. Retrieval (e.g.
on catalogues or metadata) could be done on
extensible XML structures and then forwarded to a
SQL database or archive.
16
ESA XMM SOC AMS with Tamino
AMS Archive
ATA Request

SAS ISS PPS
Other Sub- Systems
XML Catalogue Retrieval
EFT
XFTS
XFTS
XML SOAP
Internal, external Partner
XML Schema Validation
XML
XSLT Disseminator
XML Metadata Repository
XSCS
Arbritary Format (XML, non XML)
XML or other Format
17
INGESTION METHODOLOGY

• GC Proposal submitted under LET - SME
• (Leading Edge Technologies for Small and Medium
  Enterprise)
• on Archive Ingestion Methodology
• Two Phases (100 Keuro each)
• Phase 1 includes the feasibility analysis, design
  and initial demonstration via prototype of the
  archival ingestion process
• Phase 2 shall permit, from Phase 1 results, to
  implement, test and validate the procedures and
  software tools that support the ingest process.
  Interoperability between archives shall be
  demonstrated.
• Proposal is under evaluation

18
ESRIN - DEBAT STUDY

• Digital EAST Based Advanced Tools
• Support for multi file data (with hyperlinks to
  other EAST descriptions?)
• Add support for algorithms (which is inside ADA
  and was left out in the original EAST specs)
• Study of impact of nested EAST descriptions (e.g.
  detailed descriptions called automatically by the
  Interpreter only when needed)
• Add support for "namespaces". Parsers could use
  or not a specific portion of the EAST description
  according to whether it belongs to a specific
  "namespace"
• Bridges to other formats like HDF, CDF
• Add versions support to tools
• Link with UML and DEDSL
• XML support, i.e. translation to other formats
  through XML
• Full description of Envisat Products and Generic
  Data Disposition System TTC packets and history
  (Envisat plus others)
• Prototype of upgraded Interpreter and OASIS

19
ESA - OTHER ACTIVITIES

• ESTEC SSD participation at Goddard Astronomical
  Data Centre
• XML / XDF
• FITSML (upgrade of FITS to XML / XDF)
• ATV description of I/Fs using XML
• CRYOSAT S/C is using XML for I/F description