Cluster Active Archive

1
Cluster Active Archive Harri
Laakso

• Contents
• Rationale for CAA
• Quick-look plots
• CAA Database at present
• Graphical user interface
• Inventory plots
• CAA tools
• Raw data distribution
• Summary and next steps

2
Rationale for the Cluster Active Archive
Maximise the scientific return from the mission
by making all Cluster data available to the
world-wide scientific community. Ensure that
the unique data set returned by the Cluster
mission is preserved in a stable, long-term
archive for scientific analysis beyond the end of
the mission. Provide this archive as a major
contribution by ESA and the Cluster science
community to the International Living With a Star
programme.
3
Implementation of the CAA
What does this mean?

• The CAA should contain (all) the Cluster high
  resolution data.
• The data should be of the best (achievable)
  quality.
• The data should be suitable for detailed science
  investigations.
• Need to start now while expertise still
  available.
• ESA providing support to instrument teams to
  ensure this.
• In addition the CAA will hold ancillary products
  and support information including
• Auxiliary data (such as orbit and attitude)
• Survey data and plots
• CSDS data including JSOC parameters
• Documentation

4
CAA Login web-site

• For browsing the CAA database, e.g. viewing
  quick-look plots or downloading data, you need
  to login first

5
Quick-Look Plots
6
Quick-Look Plots of STAFF
7
(No Transcript)
8
Data products search
9
(No Transcript)
10
CAA Database ASPOC

• Interval over which data products exist this
  does not say anything about the completeness of
  the coverage

• Details of the dataset

11
Ion emission current, 0.5 sec resolution
12
CAA Database CIS-CODIF
O
protons
He
RPA
He
13
CAA Database CIS-HIA
14
CAA Database EDI EFW

• EDI drift velocities and electric field files
  have not yet been transferred into CEF and
  therefore not yet ingested into the system

Level 1
Level 2

• EFW the E-field products are still in spacecraft
  coordinate system, including v?B effects. The
  final transformation will be done in CAA after
  the calibrated FGM data have been ingested into
  the system

Level 2
15
CAA Database FGM STAFF
16
CAA Database PEACE

• 3DR reduced 3D dataset
• These data are provided in NM and in addition
  similar data are calculated in BM to produce a
  complete dataset
• Many other products are to be introduced in a few
  months after the definition of the formats of the
  products is completed

HEEA
LEEA
17
CAA Database RAPID
electrons
18
CAA Database Whisper

• no electron density files have not yet been
  produced
• delivery schedule for the density calibration was
  recently finished and first products should
  arrive before summer

19
Graphical User Interface Features
20
GUI - downloading
21
GUI - downloading
22
GUI - searching user profiles
23
GUI - searching user profiles
24
GUI - searching user profiles
25
GUI - searching user profiles
26
GUI - searching user profiles
27
CAA Database
28
Instrument Team Delivery Status
Team Interval Provided
ASPOC 2001-01-01 to 2002-12-31
CIS 2001-01-01 to2002-12-31
DWP 2003-12-26 to 2004-03-25
EDI 2001-02-03 to 2003-12-31
EFW 2001-02-03 to 2004-06-20
FGM 2001-01-30 to 2002-05-02
PEACE 2001-01-01 to 2002-07-01
RAPID 2001-01-01 to 2002-12-31
STAFF 2001-01-03 to 2004-12-31
WHISPER 2001-02-02 to 2002-12-31
WBD 2001-01-01 to 2002-12-31
ESOC 2001-01-01 to2004-12-31
29
Inventory Plots

• give details of the time periods for which CEF
  files have been ingested and indicate data gaps
  within these files
• enable CAA users to
• review status of products ingested in the CAA
  database
• check the availability of data for specific
  events
• help the CAA team keep track of the completeness
  of data products time periods.
• Input data sources
• CAA Gap Files (interval gt 5 minutes)
• Telemetry Modes Files (OFF, NM1-4, BM1-4 )
• JSOC Event Files(bow shock, magnetopause, north
  tail lobe...)
• Generate postscript (converted to gif for
  ingestion)

30
Inventory Plots
31
Inventory Plots
32
Inventory Plots

• Inventory plots provide data coverage information
  to assist location of intervals of interest
• Pre-generated inventory plots provided at year,
  month and day resolution.
• Make use of record level data to show true data
  availability
• Provided per-instrument, per-spacecraft.
• Accessed via quicklook type interface.
• Spacecraft telemetry and JSOC predicted science
  events also displayed

33
Inventory Plots GUI
34
CAA Standards and tools
35
CAA - Standards

• The CAA standards form the basis of a consistent
  approach to the formatting and description of the
  CAA digital data products
• The major standards development work was
  undertaken during the implementation phase by the
  data format and metadata working groups
• The Cluster Exchange Format (DS-QMW-TN-0003), has
  proved very stable and there have been no changes
  to the specification since May 2005
• The data model and data dictionary
  (CAA-CDPP-TN-0002) define the semantics used to
  describe the data
• The data dictionary has also proved very stable
  although it is continuing to be developed and
  enhanced in support of the instrument team
  product deliveries.
• Enhancements have been extensions to the existing
  set of enumerated values or additions of optional
  new attributes in support of ongoing product
  development work.
• No changes have, so far, been made, or required,
  that would affect products that have already been
  delivered.

36
CAA - Support tools

• The main CAA provided support tools currently
  provided are the CAAtools, originally developed
  by Tobias Eriksson
• Tools to allow checking of products against the
  metadata dictionary and CEF specification. Used
  by instrument teams and within CAA DMS.
• Updated for bugs and changes to the metadata
  dictionary.
• Modified to support direct operation on Gzipd
  CEF allowing data to be stored on the CAA system
  in compressed form.
• Support for indexing allowing more rapid access
  to records within the compressed (or
  uncompressed) ASCII CEF files.
• QTran and QSAS tools provided by IC QMUL
• Updated to provide improved support for CAA CEF-2
  products.
• Used to provide CDF export from the CAA web
  interface.
• QSAS provides many useful features for
  multi-instrument and multi spacecraft analysis
  and visualisation.

37
CAA QSAS support for CEF-2

C4
C2
C3
C1
1 minute
38
CAA RDM Delivery

• RDM CD-ROM delivery from ESOC ceased on 1 Jan
  2006
• CAA provides RDM (Raw Data Media) data accessible
  over the network
• DDS access will continue unchanged
• CD-ROM directory structure will be maintained
• A web based system for data retrieval has been
  put in place and is currently undergoing testing
• Requests can be made for just the portions of the
  RDM directory structure that are required, thus
  saving network bandwidth
• Automated transfers provided via wget
• ICD available on CAA website
• Details of the interface and examples are
  available on the CAA technical forum
  http//www.cluster.rl.ac.uk/caa-bin/yabb/YaBB.pl

39
Summary

• The CAA has made significant progress in the past
  year. The key milestones
• A special session at the Cluster 5th anniversary
  workshop
• Availability of the CAA for Beta testing after
  Sept 2005
• Formal opening of the CAA in February 2006
• Work has continued on the development of the core
  functionality of the system including
• Data export in NASA Common Data Format
• Support for user profiles
• Inventory plots
• Access to quicklook plots
• Raw data delivery system
• Instrument team activities have concentrated on
  refinement of the product specification and
  delivery of the data from 2001/2 and beyond.
• Instrument team delivery plans have been reviewed
  and revised.
• A CAA cross-calibration working group has been
  established.

40
Future Plans

• The major activities before the 2nd Operations
  Review (May 2007) include
• Instrument team production/delivery of 2003/2004
  data
• Routine operations, maintenance, data delivery
  and access
• Support for production and development of
  Level-3 products
• Production of EFW Level-3 products, removal of
  VxB and use of E.B0 to calculate full vector
  electric field. (Data join capability)
• On going activities of related to the cross
  calibration working group
• Josef Höök will be joining CAA in July to work on
  these tasks
• CAA ingestion of products from ESOC, CSDS, EDI
  and other sources
• Where necessary conversion to CEF format
• Production of CAA metadata information
• Support for overlapping data in ingestion and
  export
• Improved metadata (and version) information
  concatenation on export
• Development of machine accessible interface

41
Future Plans

• Improvements to web-based user interface based on
  user feedback, problem reports and RIDs
• Enhancement to user profile set-up and
  management
• Enhanced data search capability
• On-demand inventory generation (including product
  selection and filter by ingestion date, version
  and dataset number)
• Development of on-demand digital data graphical
  output (expected to be IDL based).
• Production of standard pre-generated
  multi-instrument and multi-spacecraft plots using
  CAA and accessed via quicklook browser.

42
Next Milestones
June Completion of cosmetic RID
items June Ingestion of CSDS data
2001/02/03 July On-demand inventory plotting
July/August EFW Level 3 S/W development August Pr
eliminary machine accessible interface September P
reliminary version of on-demand graphical plot
production September CAA Demonstration at
Cluster Workshop December Enhanced data search
capability 14-15th May(TBC) 2nd Operations
review Summer 07 Baseline delivery of data 2
years/year (2003 2004)
43
If you have any questions, suggestions etc,
please send a message to caateam_at_rssd.esa.int
44
CAA Management Structure
CAA Project Scientist (Harri Laakso) Technical
Manager (Chris Perry)
Cluster Project Scientist (Phillipe Escoubet)
Deputy (Matt Taylor)
Working Groups (Chairs C. Harvey A. Allen)
System Engineer (Harvey Bowen)
ESOC
JSOC
Archive Developer (Steve Esson)
Archive Developer (D. Herment)
Archive Developer (Sinead McCaffrey)
Wave Consortium
PEACE
CIS
ASPOC
EDI
RAPID
DWP
EFW
WBD
FGM
STAFF
WHISPER
Laurent Mirioni Nicole C-Wehrlin
Harald Jeszensky Klaus Torkar
Edita Georgescu Berndt Klecker
Stefan Mühlbachler Patrick Daly
Ian Bates Hugo Alleyne
Jolene Pickett
Jonny Gloag Elizabeth Lucek
Yuri Khotyaintsev Per-Arne Lindqvist Mats Andre
Alban Rochel Pierrette Decreau
Alain Barthe Henri Reme
Hina Khan Andrew Fazakerley