AIA CoreTeam Meeting

1
AIA Core-Team Meeting 20-22 April 2009 JSOC
Stuff Phil Scherrer
2
5. Data export and centers, US and beyond data
import a. Data distribution system and volume
requirements i. JSOC to world
Scherrer 20min 1440 ii.
JSOC to SAO
Scherrer/Davey 10min 1500 iii.
Elsewhere (ROB, Lancashire, )
Fleck/Boyes/Dalla 15min 1510 iv. Internal flow
Stanford to LM , and back
Hurlburt/Serafin 10min 1525 Break

1535
1555 b. JSOC (direct) data interfaces i. DRMS
system
Scherrer 20min 1555 ii. Web
interface
Summers TBD 10min 1615 iii. IDL
interfaces
Freeland 20min 1625 iv. VSO

Gurman 15min 1645 c. Data
from other instruments i. SDO HMI and EVE,
discussion
Hurlburt/Hock 15min 1700 ii. Other Discussion

Schrijver 10min 1715 d. Documentation review
and action items Scherrer/Green
15min 1725 6. Summary HMI data for AIA science
investigation Hoeksema/Scherrer 10min 1740
3
Data export and centers, US and beyond data
import a. Data distribution system and
volume requirements i. JSOC to world
http//jsoc.stanford.edu/jsocwiki/TeamMeetings
Link to SDO Pre-Ship Review(ppt)
http//hmi.stanford.edu/Presentations/SDO-PSR/25-A
IA_HMIInstScienceOps-PhilScherrer.ppt And
Link to JSOC Status shown at 2008 SDO Teams
Meeting http//hmi.stanford.edu/TeamMee
tings/Mar_2008/Proceedings/JSOC_Status_March_2008.
ppt
First page
4
HMI and AIA JSOC Architecture
5
JSOC SDP Locations at Stanford
6
JSOC Interfaces with SDO Ground System
Instrument Commands
MOC at GSFC
RT HK Telemetry (S-band)
DDS at WSC
L-0 HK files FDS products
Mission support data
RT HK telemetry L-0 HK files FDS products
Planning data
Instrument Commands
RT HK Telemetry
DDS Handshake files
Science data files (Ka-band)
AIA OPS Real-time Inst monitor and Control
AIAQL Quicklook Planning Analysis
HMI OPS Real-time Inst monitor and Control
HMI QL Quicklook Planning Analysis
TC-Segment
SDP segment
JSOC-IOC LMSAL
JSOC-SDP Stanford
JSOC-SDP Primary responsibilities Capture,
archive and process science data Additional
Instrument HS monitoring
JSOC-IOC Primary responsibilities Monitor
instruments health and safety in real-time,
24/7 Control instrument operations and generate
commands Support science planning functions
7
JSOC-SDP Major Components
8
JSOC Dataflow Rates
9
JSOC Data Volumes from Proposal
10
JSOC Data Volumes from Proposal
11
JSOC Processing Levels

• Tlm is raw telemetry files as received from SDOGS
• Level-0 is images extracted from tlm with added
  meta-data, no change to pixels
• Level-1 is cleaned up and calibrated into
  physical units in standardized form
• Level-2 is science data products
• Level-3 is higher level products or user produced
  products and are not JSOC products but may be
  archived and distributed as desired by owner

12
JSOC DCS Science Telemetry Data Archive

• Telemetry data is archived twice
• The Data Capture System (DCS) archives tlm files
  for offsite storage
• Archive tapes are shipped to the offsite location
  and verified for reading
• The Data Capture System copies tlm files to the
  Pipeline Processing System
• The Pipeline Processing System generates Level-0
  images and archives both tlm and Level-0 data to
  tape
• Only when the DCS has received positive acks on
  both tlm archive copies does it inform the DDS,
  which is now free to remove the file from its
  tracking logic

13
HMI and AIA Level-0

• Level-0 Processing is the same for HMI and AIA
• Level 0.1 Immediate Used for Ops quicklook
• Reformat images
• Extract Image Header meta-data
• Add Image Status Packet high-rate HK Packet
  (per image)
• Export for JSOC IOC Quicklook
• Level 0.3 Few minute lag - Used for quicklook
  science data products
• Add other RT HK meta-data
• Add FDS S/C info
• Level 0.5 Day or more lag Used for final
  science data products
• Update FDS data
• Add SDO roll info
• Includes final images

14
Level 1 HMI and AIA Basic Observable Quantities

• HMI and AIA level-1 levels are similar but the
  details differ (a lot).
• HMI from filtergrams to physical quantities
• 1.0 Flat field applied to enable limb fit and
  registration
• 1.5 Final product types
• 1.5q Quicklook available in 10 minutes, saved
  10 days
• 1.5p Provisional mix of 1.5q and 1.5 final
• 1.5 Final best possible product
• Products
• Continuum Intensity
• Doppler Velocity
• Line of Sight Field
• Vector Field
• AIA Filtergrams are basic product
• Quicklook and Final both produced
• Planning movies from quicklook
• Full details in development

15
Newer Processing Flow Diagrams
http//jsoc.stanford.edu/jsocwiki/Lev1Doc
Link to Cmap at diagram to http//jsoc.stanford.e
du/Cmaps/web/JSOC_SDP_Data_Flow.html On
following pages ---
16
JSOC LEVEL-0 Processing
Immediate
Level-0 HSB image Immediate or Retransmitted, perm
anent
DDS
JSOC-IOC quicklook, Temp, 5d
Level-0.1
JSOC IOC
Ground Tables
Level-0 HK ISP
DDS
Level 1.0q Flat fielded and bad pixel list
included, Temp 1d
HK 1553 Other APID Level-0
Select nearest or average
HK via MOC
Level-0.3
Command logs if needed
Level 1.5q Quicklook Observables, Temp, 5d
JSOC IOC
FDS series, temp
FDS predict data dayfiles
MOC
Few minutes lag
Level 1.5p Provisional Observables, Links to best
avail
FDS final data dayfiles
FDS series, temp
MOC
Level 1.5 Final Observables, permanent
HK 1553 APID dayfiles
Level-0.5
Level-0 HK, temp
MOC
Level 1.0 Flat fielded and bad pixel list
included, Temp 60d
SDO HK dayfiles From MOC
SDO HK lev0 temp
MOC
Day lag
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Configuration Management Control

• Capture System
• Managed by JSOC-SDP CCB after August freeze
• Controlled in CVS
• SUMS, DRMS, PUI, etc. Infrastructure
• Managed by JSOC-SDP CCB after launch
• Controlled in CVS
• PUI Processing Tables
• Managed by HMI and/or AIA Instrument Scientist
• Controlled in CVS
• Level 0,1 Pipeline Modules
• Managed by HMI and/or AIA Instrument Scientist
• Controlled in CVS
• Science Analysis Pipeline Modules
• Managed by program author
• Controlled in CVS

24
Data export and centers, US and beyond data
import a. Data distribution system and
volume requirements And b. JSOC (direct)
data interfaces i. DRMS system
ii. Web interface
http//jsoc.stanford.edu/jsocwiki/TeamMeetings
Link to JSOC Status shown at 2008 SDO Teams
Meeting http//hmi.stanford.edu/TeamMee
tings/Mar_2008/Proceedings/JSOC_Status_March_2008.
ppt
First page
25
JSOC Export

• ALL HMI and AIA data will be available for export
  at level-1 through standard products (level-1 for
  both and level-2 for HMI)
• It would be unwise to expect to export all of the
  data. It is simply not a reasonable thing to
  expect and would be a waste of resources.
• Our goal is to make all useful data easily
  accessible.
• This means we must develop browse and search
  tools to help generate efficient data export
  requests.
• Quicklook Products
• Quicklook raw images to JSOC IOC
• Quicklook Basic Products to Space Weather Users
• Prime Science Users
• JSOC will support Virtual Solar Observatory (VSO)
  access
• JSOC will also have a direct web access
• There will be remote DRMS/SUMS systems at key
  Co-I institutions
• JSOC In Situ Delivery and processing
• Special Processing at JSOC-SDP as needed and
  practical
• Public Access
• Web access for all data Special products for
  E/PO and certain solar events

26
JSOC DRMS/SUMS Basic Concepts

• Each image is stored as a record in a data
  series.
• There will be many series e.g. hmi_ground.lev0
  is ground test data
• The image metadata is stored in a relational
  database our Data Record Management System
  (DRMS)
• The image data is stored in SUMS (Storage Unit
  Management System) which itself has database
  tables to manage its millions of files.
• SUMS owns the disk and tape resources.
• Users interact with DRMS via a programming
  language, e.g. C, FORTRAN, IDL.
• The name of a dataset is actually a query in a
  simplified DRMS naming language that also allows
  general SQL clauses.
• Users are encouraged to use DRMS for efficient
  use of system resources
• Data may be exported from DRMS as FITS or other
  protocols for remote users.
• Several Remote DRMS (RDRMS) sites will be
  established which will subscribe to series of
  their choice. They will maintain RSUMS
  containing their local series and cached JSOC
  series.
• The JSOC may act as an RDRMS to access products
  made at remote sites.

27
JSOC data organization

• Evolved from FITS-based MDI dataset concept to
• Fix known limitations/problems
• Accommodate more complex data models required by
  higher-level processing
• Main design features
• Lesson learned from MDI Separate meta-data
  (keywords) and image data
• No need to re-write large image files when only
  keywords change (lev1.8 problem)
• No (fewer) out-of-date keyword values in FITS
  headers
• Can bind to most recent values on export
• Easy data access through query-like dataset names
• All access in terms of sets of data records,
  which are the atomic units of a data series
• A dataset name is a query specifying a set of
  data records (possibly from multiple data
  series)
• Storage and tape management must be transparent
  to user
• Chunking of data records into storage units and
  tape files done internally
• Completely separate storage and catalog
  databases more modular design
• Legacy MDI modules should run on top of new
  storage service
• Store meta-data (keywords) in relational database
  (PostgreSQL)
• Can use power of relational database to rapidly
  find data records
• Easy and fast to create time series of any
  keyword value (for trending etc.)
• Consequence Data records for a given series must
  be well defined (i.e. have a fixed set of
  keywords)

28
DRMS DataSeries

• A Dataseries consists of
• A SeriesName which consists of
• ltprojectnamegt.ltproductnamegt
• a sequence of Records which consist of a set of
• Keywords and
• Segments which consist of
• structure information and
• storage unit identifier
• Links which provide pointers to associated
  records in other series.
• A list of 0 or more PrimeKeys which are keywords
  sufficient to identify each record (default to
  recnum)

29
DRMS DataSeries - cont

• Data is accessed in RecordSets which are
  collections of records identified by seriesname
  and primekeys
• RecordSets are identified by a name which is
  really a query.
• Records may have versions which have the same set
  of primekey values, most recent is current
  record.
• See http//jsoc.stanford.edu/jsocwiki/DataSeries

30
Logical Data Organization
Single hmi.fd_V data record
JSOC Data Series
Data records for series hmi.fd_V
Keywords RECORDNUM 12345 Unique serial
number T_OBS 2009.01.05_232240_TAI DATAMIN
-2.537730543544E03 DATAMAX
1.935749511719E03 ... P_ANGLE
LINKORBIT,KEYWORDSOLAR_P
hmi.lev0_cam1_fg
hmi.lev1_fd_V12345
aia.lev0_cont1700
hmi.lev1_fd_V12346
hmi.lev1_fd_M
hmi.lev1_fd_V12347
hmi.lev1_fd_V
Links ORBIT hmi.lev0_orbit, SERIESNUM
221268160 CALTABLE hmi.lev0_dopcal, RECORDNUM
7 L1 hmi.lev0_cam1_fg, RECORDNUM 42345232 R1
hmi.lev0_cam1_fg, RECORDNUM 42345233
hmi.lev1_fd_V12348
aia.lev0_FE171
hmi.lev1_fd_V12349

hmi.lev1_fd_V12350
hmi.lev1_fd_V12351
hmi.lev1_fd_V12352
Data Segments Velocity
hmi.lev1_fd_V12353

Storage Unit Directory
31
JSOC Pipeline Processing System Components
Pipeline processing plan
Pipeline Operator
SUMS Disks
Processing script, mapfile List of pipeline
modules with needed datasets for input, output
PUI Pipeline User Interface
SUMS Storage Unit Management System
DRMS Data Record Management System
SUMS Tape Farm
Processing History Log
Database Server
32
Pipeline batch processing

• A pipeline session is encapsulated in a single
  database transaction
• If no module fails all data records are commited
  and become visible to other clients of the JSOC
  catalog at the end of the session
• If failure occurs all data records are deleted
  and the database rolled back
• It is possible to commit data produced up to
  intermediate checkpoints during sessions

Pipeline session atomic transaction
DRMS Server Initiate session
DRMS Server Commit Data Deregister
Analysis pipeline

Input data records
Output data records
DRMS Service Session Master
SUMS
Record Series Database
33
Data export and centers, US and beyond data
import ii. JSOC to SAO
http//jsoc.stanford.edu/netdrms/
First page
34
Remote DRMS/SUMS

• Cooperating sites run NetDRMS code which is the
  JSOC DRMS/SUMS code base.
• They maintain their own PostgreSQL database
• Remote systems can subscribe to series created
  at other DRMS sites
• Subscribed series DRMS records are synchronized
  automatically with a short lag
• SUMS Storage Units (SUs) which contain the file
  data are imported on demand to the remote SUMS
  when a non-local sunum is requested.
• JSOC will serve all and will receive data from
  some remote sites.

35
Remote DRMS Sites

• Site Location
  Contact SUMS ID
• CFA Cambridge, MA, USA Alisdair
  Davey 0x0004
• CORA Boulder, CO, USA Aaron
  Birch 0x0005
• GSFC Greenbelt, MD, USA Joe Hourclé
  0x0002
• IAS Toulouse, France
  Frederic Auchere 0x0018
• IIAP Bangalore, India
  Paul Rajaguru 0x000c
• JSOC Stanford, CA, USA Art
  Amezcua 0x0000
• JILA Boulder, CO, USA Deborah
  Haber 0x0008
• LMSAL Palo Alto, CA, USA John
  Serafin 0x0023
• MPI Katlenburg-Lindau, Germany Raymond
  Burston 0x0001
• MSSL Dorking, UK
  Elizabeth Auden 0x0020
• NSO Tucson, AZ, USA Igor
  Suarez-Sola 0x0003
• ROB Brussels, Belgium Benjamin
  Mampaey 0x001d
• Yale New Haven, CT, USA Charles
  Baldner 0x0010

36
Web Access

• JSOC page at http//jsoc.stanford.edu
• Semantics see Jsocwiki at http//jsoc.stanford.ed
  u/jsocwiki
• Syntax for code see Man Pages
• Access for data see e.g. http//jsoc.stanford.edu
  /ajax/lookdata.html
• Also links for CVS repository and trouble reports

First page
37
(No Transcript)
38
Work Remaining

• Oh, gee
• Web browsable catalog
• Better user experience
• Links to HKB
• VSO provided SU availability catalog
• Testing
• Not to mention HMI analysis code
• Testing
• Data from the Sky

39
6. Summary HMI data for AIA science investigation
http//hmi.stanford.edu/Presentations/LWS-2007-Tea
msDay/HMI_Dataproducts_Sept_2007.ppt
First page
40
Primary goal origin of solar variability

• The primary goal of the Helioseismic and Magnetic
  Imager (HMI) investigation is to study the origin
  of solar variability and to characterize and
  understand the Suns interior and the various
  components of magnetic activity.
• HMI produces data to determine the interior
  sources and mechanisms of solar variability and
  how the physical processes inside the Sun are
  related to surface and coronal magnetic fields
  and activity.

41
Key Features of HMI Science Plan

• Data analysis pipeline standard helioseismology
  and magnetic field analyses
• Development of new approaches to data analysis
• Targeted theoretical and numerical modeling
• Focused data analysis and science working groups
• Joint investigations with AIA and EVE
• Cooperation with other space- and ground-based
  projects (SOHO, Hinode, PICARD, STEREO, RHESSI,
  GONG, SOLIS, HELAS)

42
(No Transcript)
43
Primary Science Objectives

• Convection-zone dynamics and solar dynamo
• Structure and dynamics of the tachocline
• Variations in differential rotation.
• Evolution of meridional circulation.
• Dynamics in the near-surface shear layer.
• Origin and evolution of sunspots, active regions
  and complexes of activity
• Formation and deep structure of magnetic
  complexes.
• Active region source and evolution.
• Magnetic flux concentration in sunspots.
• Sources and mechanisms of solar irradiance
  variations.
• Sources and drivers of solar activity and
  disturbances
• Origin and dynamics of magnetic sheared
  structures and delta-type sunspots.
• Magnetic configuration and mechanisms of solar
  flares and CME.
• Emergence of magnetic flux and solar transient
  events.
• Evolution of small-scale structures and magnetic
  carpet.
• Links between the internal processes and
  dynamics of the corona and heliosphere
• Complexity and energetics of solar corona.
• Large-scale coronal field estimates.

44
HMI Science Analysis Plan
Magnetic Shear
45
HMI module status and MDI heritage
Intermediate and high level data products
Primary observables
Internal rotation
Heliographic Doppler velocity maps
Spherical Harmonic Time series
Mode frequencies And splitting
Internal sound speed
Full-disk velocity, sound speed, Maps (0-30Mm)
Local wave frequency shifts
Ring diagrams
Doppler Velocity
Carrington synoptic v and cs maps (0-30Mm)
Time-distance Cross-covariance function
Tracked Tiles Of Dopplergrams
Wave travel times
High-resolution v and cs maps (0-30Mm)
Research codes in use by team
Egression and Ingression maps
Wave phase shift maps
Deep-focus v and cs maps (0-200Mm)
Far-side activity index
Stokes I,V
Line-of-sight Magnetograms
Line-of-Sight Magnetic Field Maps
Stokes I,Q,U,V
Full-disk 10-min Averaged maps
Vector Magnetograms Fast algorithm
Vector Magnetic Field Maps
Vector Magnetograms Inversion algorithm
Coronal magnetic Field Extrapolations
Tracked Tiles
Tracked full-disk 1-hour averaged Continuum maps
Coronal and Solar wind models
Continuum Brightness
Solar limb parameters
Brightness feature maps
Brightness Images
46
JSOC - HMI Pipeline
Data Product
Processing
HMI Data
Internal rotation O(r,T) (0ltrltR)
Spherical Harmonic Time series To l1000
Heliographic Doppler velocity maps
Filtergrams
Mode frequencies And splitting
Internal sound speed, cs(r,T) (0ltrltR)
Full-disk velocity, v(r,T,F), And sound speed,
cs(r,T,F), Maps (0-30Mm)
Local wave frequency shifts
Ring diagrams
Doppler Velocity
Level-0
Carrington synoptic v and cs maps (0-30Mm)
Time-distance Cross-covariance function
Tracked Tiles Of Dopplergrams
Wave travel times
High-resolution v and cs maps (0-30Mm)
Egression and Ingression maps
Wave phase shift maps
Deep-focus v and cs maps (0-200Mm)
Far-side activity index
Stokes I,V
Line-of-sight Magnetograms
Level-1
Line-of-Sight Magnetic Field Maps
Stokes I,Q,U,V
Full-disk 10-min Averaged maps
Vector Magnetograms Fast algorithm
Vector Magnetic Field Maps
Vector Magnetograms Inversion algorithm
Coronal magnetic Field Extrapolations
Tracked Tiles
Tracked full-disk 1-hour averaged Continuum maps
Coronal and Solar wind models
Continuum Brightness
Solar limb parameters
Brightness Images
Brightness feature maps
HMI Data Analysis Pipeline
47
Magnetic Fields
48
Line-of Sight Magnetic Field
49
Vector Magnetic Field
Need
50
Intensity