JSOC SDP Agenda

1
JSOC SDP Agenda

• Significant Level 4 requirements
• SDP Architecture Decomposition
• Datacapture System
• Datacapture System Components
• Pipeline Processing System
• Pipeline Processing System Components
• Software Configuration Items
• Hardware Configuration Items
• Data Archive
• Data Distribution
• Performance Analysis
• Heritage
• SDP Network Design Security
• Test approach
• Implementation plan
• Deliverables hardware/software/documentation
• Development environment/CM Process
• Sustaining support pre-launch and post-launch

2
DMR JSOC SDP Requirements

• 8000.2.4 Science Data Processing, Archiving and
  Distribution
• Each SOC shall provide the necessary facility,
  software, hardware and staff to receive, process,
  archive and distribute the science data generated
  by its instruments.

My Documents\DMR_SOC_Req.ppt
3
JSOC (Stanford Science Data Processing)
Configuration
DDS
SDP
Pipeine Processing
Datacapture System
Switch
Switch
File Server
ethernet
Data Base Server
Analysis Cluster
heartbeat
Passive
Active
Disk arrays
Server
Server
Fibre Channel Switch
etc.
Disks
Tape Robot
Pipeline Processor
Tape Robot
LAN
jim\My Documents\hardware_config3.vsd
4
JSOC Data Capture System Components
lev0 module
DDS
Operator
Primary Storage Disks
Utility Libraries
GUI Data Capture Interface
FTP Disks
Record Cache
Tape Server
DSDS Data Storage Distribution System
Processing History Log
Tape Drive
Database Server
5
JDAT network
6
JSOC Pipeline Processing System Components
Pipeline processing plan
Pipeline Operator
SUMS Disks
Processing script, mapfile List of pipeline
modules with needed datasets for input, output
PUI Pipeline User Interface
SUMS Storage Unit Management System
DRMS Data Record Management System
SUMS Tape Farm
Processing History Log
Database Server
7
S/W Configuration Items

• SUMS
• DRMS
• PUI
• Event Mgr
• Science Libs
• Pipeline Util Libs
• Export Support
• Lev0
• Lev1
• Standard Products

8
H/W Configuration
9
H/W Configuration (cont)
10
Telemetry Data Archive

• Telemetry data is archived twice
• The Data Capture System archives tlm files for
  offsite storage
• Archive tapes are shipped to the offsite location
  and verified for reading
• A feedback mechanism will be established to ack
  the SDP that a tape is verified or that another
  copy needs to be sent
• The Data Capture System copies tlm files to the
  Pipeline Processing System
• The Pipeline Processing System archives tlm data
  for local storage and acks the SDP when it is
  successful
• Only when the SDP has received positive acks on
  both archive copies does it inform the Data
  Capture System to include the tlm file in the
  .arc file to the DDS, who is now free to remove
  the file from its tracking logic
• The lev0 data on the Pipeline Processing System
  is also archived

11
JSOC Data Export System
Researcher B
Space Weather
API
Script Access
Keywords Range
Researcher A
Selected Data Records
Drilldown
Overview New/Avail Statistics
General Public
DRMS
12
Performance Analysis

• AIA/HMI combined data volume 2 PB/yr 60 MB/s
• read write x 2
• quick look final x 2
• one reprocessing x 2
• 25 duty cycle x 4
• 2 GB/s (disk)
• 0.5 GB/s (tape)
• NFS over gigabit ethernet (50-100MB/s/channel) 1
  4 GB/s
• 4 8 channels per server, 5 servers (today)
• SAIT-1 native transfer rate (25-30 MB/s/drive)
  0.5 0.6 GB/s
• 10 SAIT-1 drives per library, 2 libraries (today)

13
Heritage

• MDI ground system design and implementation
• MDI production processing for 9 years
• MDI sustaining engineering for h/w and s/w
  upgrades
• MDI lessons learned folded into JSOC SDP design

14
Stanford/Lockheed Connections
Stanford
DDS
NASA AMES
LMSAL
1 Gb Private line
MOC
White Net
15
Test Approach

• Telemetry data is simulated with MDI data
  packaged in SDO VCDUs
• Various telemetry pathologies can be created to
  validate telemetry processing
• The DDS is simulated by packaging the VCDUs into
  files and injecting them into the SOC front end
  using the DDS/SOC protocol
• The lev0 processing is validated by comparing
  input and output images
• Timing analysis and throughput studies will be
  performed to evaluate h/w and s/w tradeoffs
• Instrument sunlight tests will be used to
  validate the lev1 processing
• Each s/w library and subsystem has a regression
  test suite that is run to verify each new CM
  release
• Full end to end testing is accomplished via the
  mission IT plan

16
Integrated Schedule
JSOC Science Data Processing (SDP) / DDS IT
Start Dates Delivery of Flight EGSE SDP June
2005 Prototype SDP System Ready Dec 2005 JSOC
Network Ready Dec 2006 DDS-JSOC Testing
Dec 2006 GSRT2- Science Data Processing
Test (Ka-band) Jan 2007 HMI Connectivity,
Dataflow, Retransmissions Test Feb 2007 AIA
Connectivity, Dataflow, Retransmissions Test Feb
2007 GSRT3-Mission Operations RF
Communications Test Mar 2007 GSRT4-Fully
Integrate Ground System Mar 2007 Ground
System Freeze Jan 2008 GSRT4-Launch
Readiness Test Feb 2008
17
Implementation Plan

• TBD Get from Carl

18
Deliverables

• EGSE hardware
• EGSE software
• TBD Get from Carl (from contract)

19
Configuration Management Control

• Capture System
• Managed by JSOC CCB
• Controlled in CVS
• SUMS, DRMS, PUI, etc. Infrastructure
• Managed by JSOC CCB after launch
• Controlled in CVS
• PUI Processing Tables
• Managed by HMI and/or AIA Instrument Scientist
• Controlled in CVS
• Level 0,1 Pipeline Modules
• Managed by HMI and/or AIA Instrument Scientist
• Controlled in CVS
• Science Analysis Pipeline Modules
• Managed by program author
• Controlled in CVS

20
CM with CVS
21
Prototype/Trade Study Sequence

• MDI Heritage software for EGSE Science W/S
  (currently running)
• Study to separate keywords from image data in a
  keyword Database
• MDI database optimization study
• MDI internal data set representation study
• Storage management trade study (commercial HSM
  vs. MDI heritage)
• Archive media study to replace MDI Ampex robotic
  units
• Storage Area Network (SAN) / Network Attached
  Storage (NAS) trade study
• Location of offsite media storage

22
Risks and Mitigations

• Physical location of JSOC
• - Tracking through Stanford Management
• - Rent space off-campus
• Cost risk in assumed rate of computer technology
  development
• - Advances already made
• - Wide margin of costing allowed
• Data Base Performance
• - Prototyping underway
• - Design alternatives
• - Higher performance equipment

23
Development Test Plans and Procurement Schedule

• HMI and AIA Data EGSE installed
• Prototype for I/F testing with GS Mar, 2005
  onward
• Version 2 to support flight inst. June 2005
• JSOC Capture System
• Purchase computers Summer 2006
• Support DDS testing Fall 2006
• Final system installed Spring 2007
• JSOC SDP Infrastructure, SUMS, DRMS, PUI
• Prototype testing of core system June 2005
• Fully functional Dec, 2005
• Purchase computers for JSOC Jan, 2007
• Infrastructure Operational April, 2007
• Data Product Modules Jan, 2008
• Test in IT and with DDS,MOC as called for in SDO
  Ground System schedule
• During Phase-E
• Add media and disk farm capacity in staged plan,
  half-year or yearly increments
• First two years of mission continue Co-I pipeline
  testing support

24
(No Transcript)
25
Test Approach

• Unit test
• Sub System Integration
• Regression Tests
• Performance Test
• System Integration
• I T with DDS/MOC
• On-line test results and history

OLD
26
Procurement Status

• 2004 - 2005 First 2 Years
• Procure development system with most likely
  components (e.g. tape type, cluster vs SMP, SAN
  vs NAS, etc)
• Modify pipeline and catalog infrastructure and
  implement on prototype system.
• Modify analysis module API for greater simplicity
  and compliance with pipeline.
• Develop calibration software modules.
• Complete level 0 processing code to support
  HMI/AIA instrument testing.
• 2006 - 2007 Two years prior to launch
• Complete Level-1 analysis development, verify
  with HMI/AIA test data.
• Populate prototype system with MDI/TRACE data to
  verify performance.
• Procure, install, verify computer hardware.
• Implement higher-level pipeline processing
  modules with Co-I support
• During Phase-E
• Add media and disk farm capacity in staged plan,
  half-year or yearly increments
• First two years of mission continue Co-I pipeline
  testing support

OLD