eSDO Algorithms, Visualization and Data Access

1
eSDO Algorithms, Visualization and Data Access

• Elizabeth Auden
• 27 September 2006
• HELAS Workshop
• Nice

2
What is eSDO?

• eSDO is an e-science project funded by PPARC to
  make SDO data and algorithms available to the UK
  solar community using the virtual observatory.
• eSDO Duration 36 months, October 2004 October
  2007
• SDO Launch October 2008
• RESEARCH - Phase A 1 October 2004 30 September
  2005
• IMPLEMENTATION - Phase B 1 October 2005 30
  September 2007
• Current eSDO Project Documentation
• http//www.mssl.ucl.ac.uk/twiki/bin/view/SDO

3
eSDO Workpackages

• I. Algorithms
• Code feature recognition, local helioseismology,
  global helioseismology
• Grid integration AstroGrid and JSOC
• II. Data centres
• AstroGrid interface with US JSOC data centre
• AstroGrid interface with UK (or European?) data
  centre
• III. Visualization tools
• Catalogues
• SDO Streaming tool
• On-the-fly image galleries and movies

4
eSDO Project Team

• MSSL Coronal Image Analysis
• Elizabeth Auden, project manager
• Len Culhane, PI
• Lidia van Driel-Gesztelyi
• Mike Smith
• RAL Event and Feature Recognition
• Richard Harrison
• Andrzej Fludra
• Vito Graffagnino
• Sheffield Local Helioseismology
• Mike Thompson
• Sergei Zharkov
• Birmingham Global Helioseismology
• Yvonne Elsworth
• Bill Chaplin
• Thierry Toutain

US Links JSOC, Joint Operations Centre Stanford
University Lockheed Martin
5
AlgorithmsCode

• Feature Recognition / Event Analysis
• Coronal loop detection, MSSL
• Non-linear magnetic field extrapolation, MSSL
• Helicity computation, MSSL
• Small event detection, RAL
• CME dimming region recognition, RAL
• Local Helioseismology
• Perturbation map generation, Sheffield
• Subsurface flow analysis, Sheffield
• Local helioseismology inversions, Sheffield
• Global Helioseismology
• Mode parameters analysis, Birmingham
• Mode asymmetry analysis, Birmingham

Solar image from TRACE mission. Coronal loops
marked in blue.
6
AlgorithmsGrid Integration

• JSOC
• SDO pipeline at Stanford University / Lockheed
  Martin
• eSDO algorithms deployed as pipeline modules
  (automated or user-run)
• Data will be obtained locally from JSOC archive
• AstroGrid CEA
• UK hosted algorithms hosted as web services
• Accessible through AstroGrid workflows
• Data transferred from MySpace, remote data centre
  or users local storage
• SolarSoft
• Algorithm C modules wrapped in IDL
• Distribution through MSSL SolarSoft gateway
• Data transferred to algorithm from users local
  storage

7
AlgorithmsAstroGrid Deployment

• 9 algorithms have been deployed as CEA
  applications

8
Data CentresUS Data Centre

• Primary US Data Store
• Files stored in SUMS (Storage Unit Management
  System) at Stanford University with metadata in
  distributed DRMS (Data Resource Management
  System)
• DSA / DRMS integration for grid data access
  investigations
• VSO (Virtual Solar Observatory) primary US access
• AstroGrid / VSO Interface
• VSO queries in AstroGrid workflows through STAP
  (Simple Time Access Protocol) web service
• US / UK Network Latency Tests
• Test expediency of file transfer mechanisms
• Test between Stanford, MSSL, UCL, RAL

9
Data CentresPossible UK Data Centre

• Storage
• 30 TB disc cache
• 60 day rolling cache HMI/AIA
• True cache for HMI / AIA
• DRMS / SUMS
• Transfer
• Export formats from JSOC
• Data search DSA
• Data transfer - CEA
• Test installation MSSL

Investigation of DRMS / SUMS
10
Data CentresPossible European Data Centre
TLAs DRMS Data Resource Management System
(JSOC) SUMS Storage Unit Management System
(JSOC) DSA DataSet Access (AstroGrid)

• EU Framework 6 support for European
  helioseismology data and algorithms
• Possible full mirror of SDO data in Europe?
• DSA / DRMS interface for AstroGrid access to full
  mirror of SDO data in Europe?

DRMS
DRMS
DRMS
SUMS
SUMS
US
Europe
DSA
UK datasets
UK
11
Data Centres eSDO / Solar-B Prototype system
perspective
(Local)
XML
AstroGrid HelioScope (Local)
STAP service
URLs
UKSSDCADS file system
Note AstroGrid components (JES, DSA, MySpace,
Workbench) may be located anywhere.
12
Data Centres User access through AstroGrid
AstroGrid workbench STAP service access through
HelioScope, DSA access through the Task Launcher.
Task Launcher locate DataSet Access components,
submit ADQL queries
STAP service will point to URLs in UK, US or
Germany. DSA will integrated with DRMS (Data
Resource Management System) in UK or Germany.
HelioScope solar data results from Simple Time
Access Protocol services.
13
Visualization ToolsCatalogues

• Small Event / CME Dimming catalogue
• Generated from continuously running of small
  event detection / CME dimming region recognition
  algorithms
• Tables of event / CME statistical information
• Likely to be generated in UK
• Helioseismology Mode Parameters catalogue
• GONG month of data generated from mode parameters
  analysis algorithm
• Likely to be generated in US

Catalogues can be accessed through ADQL
(Astronomical Data Query Language) query to DSA.
14
Visualization ToolsSDO streaming tool

• User Facilities
• Pan in space P
• Zoom in space P
• Pan in time
• Zoom in time
• Data Products
• LoS Magnetogram
• HMI continuum map
• AIA images
• Future Work
• FFT / wavelet analysis for coronal wave
  investigations

15
Visualization ToolsOn-the-fly Image Galleries
and Movies

• Thumbnail Extraction P
• Extract thumbnails in US from HMI and AIA science
  products
• Image Gallery Generation P
• User specifies start time, stop time, cadence,
  data product
• Image gallery generated on the fly in web browser
• Movie Generation
• User specifies start time, stop time, cadence,
  data product
• MPEG movie generated on the fly

Find data with AstroGrid HelioScope. Use PLASTIC
to load data into visualization tools.
16
eSDO and the Virtual Observatory

• eSDO has links with AstroGrid, VOTech, VSO, and
  the IVOA.

17
Future Work

• Phase B implementation development now in
  progress
• Summer / Autumn 2006
• Algorithm integration with AstroGrid test
  integration with JSOC
• Unit and science testing for algorithms
• Add STAP service client to visualization tools
• Add movie functionality to streaming tool
• Investigate DRMS integration with DSA
• 2007
• Completed algorithms
• Full algorithm integration with JSOC and
  SolarSoft
• Coronal wave analysis in SDO streaming tool
• Work with HELAS to explore European SDO data
  centre
• Catalogues

18
Further Information

• Links
• eSDO http//www.mssl.ucl.ac.uk/twiki/bin/view/SDO
  
• Astrogrid http//www.astrogrid.org
• VOTech http//wiki.eurovotech.org/bin/view/VOTech
  /
• IVOA http//www.ivoa.net
• Software to try
• AstroGrid Workbench http//www2.astrogrid.org/des
  ktop/index
• HelioScope included with workbench
• eSDO Streaming Tool http//www.mssl.ucl.ac.uk/twi
  ki/bin/view/SDO/StreamingTool
• Contacts
• eSDO Elizabeth Auden, eca_at_mssl.ucl.ac.uk
• Astrogrid Solar Science Silvia Dalla,
  s.dalla_at_manchester.ac.uk