Global Auroral Imaging Access

1
Global Auroral Imaging Access
2
GAIA An International Virtual Observatory
for Auroral Remote Sensing Data
Eric Donovan, Mikko Syrjäsuo, Emma Spanswick,
Steve Marple, Noora Partamies, Stephen Mende,
Harald Frey, Glynn Germany, Joran Moen, Farideh
Honary, Ingrid Sandahl, and Aaron Roberts.
Presented by Eric Donovan at GSFC
September 8, 2006
3
Best Case Scenario
GeoSpace Data Environment
VITMO
GAIA
VSO
CDAWEB
MADRIGAL
VMO/U
VSPO
VGO
SSDP
ASTROGRID
VMO/G
SPIDR
4
Best Case Scenario
GeoSpace Data Environment
VITMO
GAIA
VSO
CDAWEB
MADRIGAL
VMO/U
Numerous VOs with well thought out and
complementary mandates, each carrying out a
well-defined task, all interacting automatically
with each other and the world.
VSPO
VGO
SSDP
ASTROGRID
VMO/G
SPIDR
5
Worst Case Scenario
GeoSpace Data Environment
VITMO
GAIA
VSO
CDAWEB
MADRIGAL
VMO/U
VSPO
VGO
SSDP
ASTROGRID
VMO/G
SPIDR
6
Worst Case Scenario
GeoSpace Data Environment
VITMO
GAIA
VSO
CDAWEB
MADRIGAL
VMO/U
Numerous VOs with overlapping mandates, all doing
more or less the same thing, all independent of
one another, and none doing it well.
VSPO
VGO
SSDP
ASTROGRID
VMO/G
SPIDR
7
Where are we?
Worst Case
Best Case
How are we doing?
Numerous VOs with overlapping mandates, all doing
more or less the same thing, and none doing it
well.
Numerous VOs with well thought out and
complementary mandates, each carrying out a
well-defined task, all interacting automatically
with each other and the world.
8
Some Thoughts on Spanning VO-Space
Horizontal, Vertical, and Super Observatories

CGSM
MIRACLE
THEMIS
Oulu
SPEARS
BAS
AGI
ASI
MSP
RIO
MAG
D-SONDE
HF RADAR
VHF RADAR
VHF

9
Some Thoughts on Spanning VO-Space
Horizontal, Vertical, and Super Observatories

CGSM
MIRACLE
THEMIS
Oulu
SPEARS
BAS
AGI
ASI
MSP
RIO
MAG
D-SONDE
HF RADAR
Vertical Observatory
VHF RADAR
VHF

10
Some Thoughts on Spanning VO-Space
Horizontal, Vertical, and Super Observatories

CGSM
MIRACLE
THEMIS
Oulu
SPEARS
BAS
AGI
ASI
MSP
RIO
MAG
D-SONDE
HF RADAR
SuperDARN
VHF RADAR
VHF

11
Some Thoughts on Spanning VO-Space
Horizontal, Vertical, and Super Observatories

CGSM
MIRACLE
THEMIS
Oulu
SPEARS
BAS
AGI
ASI
MSP
RIO
MAG
D-SONDE
HF RADAR
VHF RADAR
VHF

12
Say we wanted to work towards Super-VO
We need to work together with SuperDARN, VGMO
and/or SuperMAG, and other horizontal
observatories to develop outputs that are easy
to integrate.
Be aware of SPIDR, MADRIGAL, CEDAWeb, and other
major initiatives that have not been able to
close this problem despite tens of millions of
dollars of dedicated resources..
13
Where does GAIA fit in?

CGSM
MIRACLE
THEMIS
Oulu
SPEARS
BAS
AGI
ASI
GAIA
MSP
RIO
MAG
D-SONDE
HF RADAR
VHF RADAR
VHF

14
What are we trying to accomplish?
15
What is GAIA?
Numerous overarching programs with overlapping
mandates - most with no resources to speak of.
IPY, eGY, ILWS, IHY, CAWSES, etc
ICESTAR
Project X
Project Y
Project Z
Project U
AON
GLORIA
EoI P
EoI Q
EoI N
GAIA
GAIA is a Virtual Observatory for integrating
data from the world-wide networks of optical
instruments and riometers. The overarching theme
is to provide quick and scientifically meaningful
access to data that is used primarily to specify
ion and electron precipitation. GAIA will exist
regardless of the larger programs, but is a
natural element of many of these, consistent with
their stated philosophies, brings together
researchers from (at present) seven nations, and
represents meaningful new work capabilities.
16
What is GAIA ???
User
Public Interface
GAIA
GET INFO
SRI
MIRACLE
Others?
NORSTAR
AGI
THEMIS
SPEARS
GAIA is a public interface (set of tools), to
browse summary data, data availability, and
retrieve data from a worldwide network of ASIs,
MSPs, and Riometers operated by a collaborative
and international group of researchers.
17
What is GAIA?
18
What is GAIA?
All VxOs have web-based interfaces between users
and Data bases. The GAIA interface will have
three elements (retrieval, overlay, and
browsing). Currently, it consists only of a front
page and browser being developed by Emma
Spanswick and tested by Noora Partamies. Steve
Marple of the UK will develop the prototype data
retrieval system based on the Lancaster MIA
system (projected completion February 2007).
Glynn Germany of the US will develop a system for
overlay which will allow for graphical comparison
(via maps, keograms, etc) of quantitative
intensities from different platforms (starting in
2007?).
19
What is GAIA?
For Monte Python aficionados Without data GAIA
would be like a cheese shop without cheese. Our
objective is to include data from every program
that carries out remote sensing of auroral
precipitation. We have loose agreements with
most(??) providers of optical (ground-based and
synoptic satellite-based) and riometer auroral
data. Our objective is to develop a formal data
sharing agreement by February 2007 (at or for the
ICESTAR meeting in Helsinki). Data providers are
asked to create (or allow us to create) summary
keograms and thumbnails according to rigid
specifications of dimension and filename. At a
later time data providers will be asked to stage
their full-resolution data so that the data
retrieval system can access it (a lot of work
needs to be done before we get to that point).
20
What is GAIA?
GAIA has a web-based interface between clients
and various data-bases and ultimately the data
retrieval system. GAIA has a set of core
capabilities that are in effect its brain. The
query-able relational data base (currently being
developed by Mikko Syrjäsuo of Calgary) contains
meta- and summary data that is used by the
browser and is updated on ingestion of summary
images (currently we have more than 8,000,000
summary images). The data provider interface
(currently just Mikko when he has time but to be
automated) will ingest summary images and
metadata from providers. The data retrieval and
distribution system will be developed by Steve
Marple of the UK.
21
What is GAIA?
The web-based interface is currently just a set
of scripts based on aurora.phys.ucalgary.ca. For
GAIA to be successful, these must be available at
all times, and from everywhere. Further,
performance should not be limited by the
The core capabilities are what providers and
clients interact with. These capabilities will be
mirrored in at least one location (Kirsti
Kauristie will lead that effort for FMI) and
possibly others (ie., Huigen Yang of the Polar
Research Institute in Shanghai).
22
(No Transcript)
23
Virtual Observatory Components
Distributed databases accessed through a single
GRID node
Data Visualization
Format Conversion
Data Acquisition
Location Discovery
A concept for the electronic Geophysical Year
(eGY) initiative
Taken without permission from V. Papitashvilli
24
Declaration of the eGY?
Article 1 Data access Earth system data and
information should be made available
electronically with interoperable approaches
that facilitate open access. Article 2 Data
release Owners, custodians and creators of Earth
system data should work together to share
their digital information with the world
community, though in a manner that respects
propriety requirements and security
constraints. Article 3 Data discovery Providers
and users of Earth system data and information
should share descriptions of structure, content,
and contexts to facilitate the discovery of
relationships within and between information
resources. Article 4 Data persistence Past and
present data and information about the Earth
system should be preserved and sustained in
accessible formats that are both software and
hardware independent for use today and in the
future. Article 5 Data rescue Effort should be
made to identify and rescue critical Earth system
data and ensure persistent access to
them. Article 6 Common standards and
cooperation Standards for interoperability
should be identified, created and implemented
through international collaboration. Article 7
Capability building Communities with advanced
information technology and communications
capabilities should contribute to developing
such capabilities in other communities to reduce
the digital divide. Article 8 Education and
public outreach Students, scientists,
decision-makers, and the public should be
informed about and be enabled to contribute to
the discovery of Earth system phenomena that
impact human survival.
Declaration for a Geoscience Information Commons
25
Are we operating in accordance with the
Declaration of the eGY?
Article 1 Data access Earth system data and
information should be made available
electronically with interoperable approaches
that facilitate open access. Article 2 Data
release Owners, custodians and creators of Earth
system data should work together to share
their digital information with the world
community, though in a manner that respects
propriety requirements and security
constraints. Article 3 Data discovery Providers
and users of Earth system data and information
should share descriptions of structure, content,
and contexts to facilitate the discovery of
relationships within and between information
resources. Article 4 Data Persistence Past and
present data and information about the Earth
system should be preserved and sustained in
accessible formats that are both software and
hardware independent for use today and in the
future. Article 5 Data rescue Effort should be
made to identify and rescue critical Earth system
data and ensure persistent access to
them. Article 6 Common standards and
cooperation Standards for interoperability
should be identified, created and implemented
through international collaboration. Article 7
Capability building Communities with advanced
information technology and communications
capabilities should contribute to developing
such capabilities in other communities to reduce
the digital divide. Article 8 Education and
public outreach Students, scientists,
decision-makers, and the public should be
informed about and be enabled to contribute to
the discovery of Earth system phenomena that
impact human survival.
Article 3 Data discovery Providers and users
of Earth system data should share descriptions of
structure, content, and contexts to facilitate
the discovery of relationships within and between
information resources. - At the heart of GAIA
is a relational database designed
specifically to facilitate the exploration of
natural relationships between data by users in
an easy and rapid fashion. - The GAIA Data
Providers Handbook is a living document
providing explicit instructions to data
providers vis-à-vis the inclusion of
appropriate meta-data in the GAIA relational
database. The nature of the meta-data will
evolve over time in response to user needs and
to reflect ongoing efforts of data providers. -
The overarching objective of GAIA is to
facilitate discovery!
26
Are we operating in accordance with the
Declaration of the eGY?
Article 1 Data access Earth system data and
information should be made available
electronically with interoperable approaches
that facilitate open access. Article 2 Data
release Owners, custodians and creators of Earth
system data should work together to share
their digital information with the world
community, though in a manner that respects
propriety requirements and security
constraints. Article 3 Data discovery Providers
and users of Earth system data and information
should share descriptions of structure, content,
and contexts to facilitate the discovery of
relationships within and between information
resources. Article 4 Data Persistence Past and
present data and information about the Earth
system should be preserved and sustained in
accessible formats that are both software and
hardware independent for use today and in the
future. Article 5 Data rescue Effort should be
made to identify and rescue critical Earth system
data and ensure persistent access to
them. Article 6 Common standards and
cooperation Standards for interoperability
should be identified, created and implemented
through international collaboration. Article 7
Capability building Communities with advanced
information technology and communications
capabilities should contribute to developing
such capabilities in other communities to reduce
the digital divide. Article 8 Education and
public outreach Students, scientists,
decision-makers, and the public should be
informed about and be enabled to contribute to
the discovery of Earth system phenomena that
impact human survival.
Article 6 Common standards and cooperation
Standards for interoperability should be
identified, created and implemented through
international collaboration - This will be
achieved for GAIA by the joint efforts of the IPY
AON and GLORIA projects.
27
What are the building blocks of GAIA ???
Deep Links Formulating URLs for data from
other programs.
NORSTAR-WEB summary browser (parent) creates a
dependent window that is driven in parallel to
the parent window, utilizing infrastructure
developed and maintained by another group (in
this case LANL).
28
What are the building blocks of GAIA ???
Agreed upon summary data.
Thumbnails - 1.5 kBytes
Keograms - 15 kBytes
For each day there will be one 24 hour keogram
(100 X 720 pixels) and (where appropriate) at
most 1440 thumbnail images. Note that as GAIA
accesses these summary images by formulating a
URL it absolutely does not matter whether those
summary images reside in Calgary.
29
What are the building blocks of GAIA ???
GAIA Database Management System (DBMS)
Data Catalogue
Meta Data
Summary Data
keograms Mosaics Images time series
station info instrument mode content summary data
data availability data location data policy meta
data summary data
This is the only element of GAIA that must reside
somewhere
30
What are the building blocks of GAIA ???
GAIA Database Management System (DBMS)
Example

Query - all imagers with Project
THEMIS allow the user to stack THEMIS
keograms Query - all imagers within XXkm of
Gillam allow users to view data relevant to
different scale sizes
scalability
Allows us to put any kind of interface on the
front end with amazing functionality
31
How Does it Work Version 1
32
Two Weeks One Instrument
33
One Instrument One Day
34
One Instrument One Day
35
All Instruments One Day
36
GAIA Science Center
Calibration Resources
Global Injection Identification and Tracking
Data Overlay
Identification of Auroral Boundaries
Computer Vision Pattern Recognition
37
GAIA Science Center
Calibration Resources
Global Injection Identification and Tracking
Data Overlay
Identification of Auroral Boundaries
Computer Vision Pattern Recognition
38
Work Necessary For Your Project to Be in GAIA
GAIA V1.0
Image (ASI Riometer) Components Summary
Browser

• Need to create a set of images compatible with
  the GAIA browser
• 100x100px thumbnails (grayscale)
• 720x200 daily keograms/riograms
• (Note GAIA creates two week summary
  plots)

Database Information

• Directory tree structure
• Filenaming conventions
• Imager Information
• Lat / Lon
• Filter
• Exposure

Ability to formulate URL
Ability to search based on filter, location,
content, etc.
39
Who is GAIA?
GAIA Working Group
GAIA Steering Committee
GAIA Membership

• Eric Donovan
• Kirsti Kauristie
• Stephen Mende
• Joran Moen
• Farideh Honary
• Ingrid Sandal

• Emma Spanswick
• Mikko Syrjäsuo
• Brian Jackel
• Steve Marple
• Noora Partamies
• Sanna Mäkinen
• Dirk Lummerzheim
• Andrew Kavanagh
• Dag Lorentzen
• Fred Sigernes
• Harald Frey
• Glynn Germany

• GSC GWG
• Kazuo Shiokawa
• Hans Stenbaek-Nelson
• Al Weatherwax
• Rick Doe
• Marton Blixt
• Matthieu Muerant
• Trond Trondsen
• Jim Wild
• Martin Connors
• Fokke Creutzberg
• Leroy Cogger
• Nicolai Ostgaard
• Scott Palo

Data sharing agreements Data policy Rules of the
Road All Funding Relationship to
agencies Relationship to other programs Work
package definition Work package management
Work package definition Work package
execution Scripting Data Formats Mirror
Sites Legacy Progect
Data provision Liason to other VxOs Beta-testing L
egacy Project
GAIA Legacy Project

• Brian Jackel
• Fred Sigernes
• FMI rep
• UAGI rep

Identify recover data at risk for loss
40
GAIA Milestones
Summary Browser

• three interfaces
• keogram driven
• two week summary
• all sites (single day)
• query data availability
• more interfaces
• dynamic plots
• mosaics
• real-time
• retrieve (CDF) data on demand
• mirror sites
• Orientation
• Mapping
• Cloud Cover
• Classification
• Mosaicing
• Keogram Plotter

V1.X
Sept. 1, 2005
Data Access
Summary Browser
V2.X
Jan. 1, 2007
Data Access
V3.X
Mar. 1, 2007
Software Package
ongoing
Red THEMIS Deliverable
41
GAIA Milestones
Summary Browser

• three interfaces
• keogram driven
• two week summary
• all sites (single day)
• query data availability
• more interfaces
• dynamic plots
• mosaics
• real-time
• retrieve (CDF) data on demand
• mirror sites
• Orientation
• Mapping
• Cloud Cover
• Classification
• Mosaicing
• Keogram Plotter

V1.X
Sept. 1, 2005
Data Access
Summary Browser
V2.X
Jan. 1, 2007
Steve Marple is coming to Calgary for October
2006 to March 2007. His primary objective is to
integrate MIA with GAIA MIA will produce CDF
files on demand frolm the GAIA interface.
Data Access
V3.X
Mar. 1, 2007
Software Package
ongoing
Red THEMIS Deliverable
42
GAIA Milestones
Summary Browser

• three interfaces
• keogram driven
• two week summary
• all sites (single day)
• query data availability
• more interfaces
• dynamic plots
• mosaics
• real-time
• retrieve (CDF) data on demand
• mirror sites
• Orientation
• Mapping
• Cloud Cover
• Classification
• Mosaicing
• Keogram Plotter

V1.X
Sept. 1, 2005
Data Access
Summary Browser
V2.X
Jan. 1, 2007
Version 1 is up and running. We have now
gt8,000,000 summary images in GAIA. His is growing
rapidly. Please have a look. Please comment and
criticize.
Data Access
V3.X
Mar. 1, 2007
Software Package
ongoing
Red THEMIS Deliverable
43
(No Transcript)