Searching for Data with the Virtual ITM Observatory

1
Searching for Data with the Virtual ITM
Observatory

• D. Morrison, M. Weiss, R. Daley, L. Immer, C.
  Colclough, R. Holder, J. Jen, D. Patrone, M.
  Hashemian, P. Meckel, M. Potter, R. Barnes, S.
  Nylund, J.-H. Yee, and E. Talaat
• (Johns Hopkins University, Applied Physics
  Laboratory 11100 Johns Hopkins Rd. Laurel MD,
  20723 240-228-4172)
• J. Russell
• (Hampton University)
• R. Heelis
• (Univ. Texas at Dallas)
• J. Kozyra
• (Univ. of Michigan)
• D. Bilitza, R. McGuire, and R. Candey
• (NASA Goddard Spaceflight Center)
• P. Fox
• (HAO/NCAR)

2
Primary Tasks for Mission Data Center or Virtual
Observatory

• (1) describe their data holdings and resources
• (2) discover what data sets are available for
  solving the users particular science problem
• (3) provide a means to access this data or
  resource
• (4) provide tools to help personnel use the data
  that they have found.

3
What is a VxO?

• A Virtual Observatory (VO) is a suite of
  software applications on a set of computers that
  allows users to uniformly find, access, and use
  resources (data, software, document, and image
  products and services using these) from a
  collection of distributed product repositories
  and service providers. A VO is a service that
  unites services and/or multiple repositories.
• An observatory is a location used for observing
  terrestrial and/or celestial events.
• A virtual observatory is a system that allows an
  observer to work as if they were at the
  observatory.
• The VO does this typically by allowing the
  observer to access multiple data files
  (observations) from multiple sources of the
  object of interest.
• The VO allows access to different instruments
  to study a given system or phenomena.
• VitmO ties together multiple data sets from
  different satellites and ground stations to
  create the appearance of having many different
  types of instruments observing the same phenomena
  from different locations.

4
Virtual ITM Observatory

• The Virtual ITM Observatory (VITMO) is one of the
  recently awarded domain specific NASA virtual
  observatories (VxO).
• VITMO will focus on data covering the ITM region
  as well as providing connections to the energy
  drivers for this region (principally solar and
  magnetospheric).
• VITMO is focused on the interdisciplinary ITM
  user rather than the instrument team user.

5
ITM is Diverse in Observables
An ITM VO must facilitate correlative study
between multiple parameters from multiple sources
and cross-discipline studies.
6
ITM Domain Differences

• Data covering the ITM region has a number of
  differences from data in other domains.
• The ITM region is observed by ground based remote
  sensing instruments, satellite based remote
  sensing instruments, and in-situ satellite
  instruments.
• There are external drivers in solar radiation and
  the solar wind and magnetospheric particle
  inputs.
• A Virtual Observatory that covers the ITM region
  needs to deal with the large diversity of data
  types in the study of this region.
• A heterogeneous data format environment also
  exists here.
• Solar imagery is typically in FITS files, ground
  based data are typically ASCII or NetCDF, in-situ
  satellite data are commonly in CDF format, remote
  sensing satellite data are typically HDF, NetCDF,
  or CDF.
• Non-NASA data sources (i.e. NSF, NOAA, DoD) are
  important contributors to the ITM domain.

7
Discovery of Data

• With the introduction of imaging instruments in
  ITM physics the data volume has exploded. There
  are now terabytes of data in the ITM domain.
• This talk will focus on (2) discover what data
  sets are available for solving the users
  particular science problem.
• VITMO uses an approach where external databases,
  calculated events, or other externally generated
  event lists can be used to restrict data searches
  to precise periods of relevance to the user study.

8
The Ability to Find the Appropriate Data Will Be
a Key Requirement of a VITMO
Increasingly satellites in the ITM community will
rely on remote sensing instruments TIMED, DMSP,
NPOESS
9
Typical Researcher Workflow
http//swdcwww.kugi.kyoto-u.ac.jp/index.html

• Workflow
• Identify periods of interest (onset of storm,
  etc)
• Go to various websites to download data from
  different instruments
• Plot data to identify overlaps in time and space
  of data sets
• If appropriate data not found repeat loop with
  different time period

http//www.srl.caltech.edu/ACE/ASC/
10
Download Data from Different Websites
11
Check Data from Different Satellites for
Coincidences
April 2 UT
Dec 18 UT
12
Metadata Limits the Searches

• Typical metadata does not allow the level of
  search that our typical researcher would like to
  make.
• Typical metadata includes
• descriptive metadata includes program, satellite
  and instrument information, as well as data
  details like observed phenomena or physical
  parameters measured, units of measure, time and
  spatial coverage (at a broad level only),
  processing level
• structural metadata defines the details for data
  access and retrieval if selected by the use
• Typical metadata does not include
• Time and location information at the pixel level

13
VITMO Search Architecture
VITMO
VITMO Metadata Catalog
ACE
SuperDARN
TIMED
CDAWeb
VSTO
14
Solar Geophysical Indices Database

• The TIMED program maintains a database of Solar
  Geophysical Indices that the ITM community
  decided were relevant.
• VITMO queries the TIMED Oracle database directly
  pulling the SGI parameters of interest.
• We then build windows when all conditions are
  met.

15
Coverlet Windows
Event A 3ltKplt4
Event B Solar Wind Speed gt 400 km/s
Event C Bzlt0
Resultant Timeline
16
What will VITMO search capability mean to users?

• By taking on a more global view of data, VITMO
  can help the researcher understand the
  connectedness between the different data sets.
• Example - storm studies - taking the Halloween
  2003 superstorm as a case study.
• During this event the following things happened
  (1) 3 active regions (ARs) produced a total of at
  least 124 soft X-ray flares and more than 60
  coronal mass ejections (CMEs) between Oct. 18 and
  Nov. 5, 2003. (2) 2 CMEs impacted Earth head-on
  producing huge geomagnetic storms.
• Today without VITMO
• A researcher may start with LASCO images and ACE
  solar wind data over the broad time range in
  October and November 2003. If the researcher was
  interested in auroral impacts from the storm they
  may have looked for time periods when Kp was
  between 3 and 9 and Bz was negative.
• They might have looked for Polar and GUVI imagery
  or auroral conductivity maps during those time
  periods.
• All of this work would have involved finding and
  navigating numerous web sites that provide data,
  indices, and other information for the specific
  time period.
• Now with VITMO
• The researcher will be able to select the needed
  LASCO images and ACE solar wind data.
• They will also be able to select auroral data
  during the time periods when Kp is between 3 and
  9 and Bz was negative.
• The VITMO, knowing that the researcher was
  interested in auroral data, would have
  automatically presented GUVI and Polar instrument
  products as appropriate.
• It would then suggest data readers, appropriate
  for the data format, and tools that the
  researcher could use to work with or analyze that
  data.

17
Data Search Before VITMO
18
Select Coverage
19
Select Physical Parameters
20
Restrict Coverage
21
Select Products for Download
22
Coverage Windows
23
Files Are Packaged Up For Download
24
Applicability to ITM Research

• Example Using TIMED GUVI data the researcher
  has identified ring current aurora (dayside
  detached arcs, nightside detached aurora, proton
  spots, etc).
• It is noted that in the cases examined certain
  geophysical conditions occur.
• Search for all data (from other instruments as
  well) under those conditions to identify
  occurrence frequency.

25
Overcoming the Limits of Our Metadata
GUVI 1356 Å
Plasma bubbles
Limb Scan
74o Inclination
Cross-track scan perpendicular to orbit
Disk Scan
Equatorial arcs
26
Calculated Databases

• VITMO will employ special coincidence calculators
  that know the viewing geometries of the various
  instruments.
• The TIMED coincidence calculator is an example of
  a tool which when converted into a web service
  can be used to augment existing data querying.
• The TIMED coincidence calculator has recently
  been imbedded into the TIMED query system.
• These calculators will operate as web services
  inside the VITMO query system (or by outside
  query systems).
• These calculators generate metadata for the
  coincidence of interest which is used by the
  catalog system to answer your query.
• The metadata generated by this service is
  discarded after it is used hence virtual
  metadata.

27
Not Plotting Tools Imbedded Into Query
28
Coincidence Calculator
29
VITMO Search Architecture
VITMO
VITMO Metadata Catalog
SuperDARN
TIMED
CDAWeb
VSTO
ACE
30
Enhanced Query

• With the inclusion of virtual metadata sources a
  user can go to VITMO to ask During the last 6
  months where are GUVI EDP products over Millstone
  Hill radar when Kp gt 3.
• The coincidence calculator will calculate when
  the GUVI product overlaps spatially and
  temporally with Millstone Hill.
• This is combined with when Kp gt 3.
• Query is sent to VSTO or MADRIGAL to find
  appropriate radar or other products.
• All data products returned for download.

31
Enhanced Search Results

• Addition of virtual metadata capabilities allows
  pre-selection of data for analysis.
• VITMO search will employ calculator services that
  allow us to deliver results from searches such as
  What GUVI data that contains 23-01 MLT during
  active geomagnetic conditions?

32
Products of Searches Are Windows

• Products of searches are windows of time.
• Future products of searches may be products of
  time and space.

33
Find Data During Events

• Capability exists to search for data from a
  library of events
• Solar flares
• Geomagnetic storms
• User generated conditions
• Model data comparisons
• Identify where GUVI electron density profiles
  deviate from International Reference Ionosphere
  calculations in latitudes between 30 mlat by
  more than a factor of two.
• Now find all DMSP SESS ionospheric data at the
  same time and location.

34
VITMO Search Architecture
VITMO
VITMO Metadata Catalog
SuperDARN
TIMED
CDAWeb
VSTO
ACE
35
Conclusion

• Addition of SGI databases, on-the-fly calculated
  databases (virtual metadata generators), and
  event listings greatly enhance the search
  capabilities far beyond the limitations of the
  present metadata available to describe products
  in our domain.
• The VITMO prototype is available at
  HTTP//vitmo.jhuapl.edu.
• VITMO will be operational at the end of 2007.