Title: ALMA Computing IPT Science Software Status Update
1ALMA Computing IPTScience Software Status Update
- Gianni Raffi, ESO
- ALMA Science Advisory Committee
- 15-16 September 2006
21.07 Computing Summary Schedule
(Data from IPS as _________)
Time Now
Array Software gtgt
Observatory Software gtgt
Computing IPT Management and Science Software
Requirements
High Level Analysis and Software Engineering
Common Software and Archive
Executive, Correlator and Control SW
Schedule, Observation Prep, Offline, and
Telescope Calibration
Software Integration and Test
3Requirements Management
- Goal
- Evaluate development progress with respect to
requirements - Provide reporting.
- gt Done via Central database (DOORS) with
requirements derived from all sources (science,
technical, operational) that allows us to - Manage requirements (addition, editing, deletion,
prioritization, review) - Assess the status of software development
(reporting) - Analyze the impact of requirements changes
- Coordinate requirement tracking for all computing
subsystems. - Complete functional team work and release
development with integration tests (by ITS team) - Validate implementation with user tests (by SSR
subsystem scientists) following each release.
4SSR Testing of Science Software
- Extracted from D.Shepherds page on TWiki at
- http//almasw.hq.eso.org/almasw/bin/view/Usertests
/WebHomeTest_Plans - Requirements available on SSR TWiki (unless
explicitly mentioned below) - All functional subsystems shown for completeness
(except ACS framework and ObOps) - EXEC
- R3.1 - Executive internal pre-test complete.
- R4.0 - user test scheduled to begin October 20.
- Scheduler
- R3.1 - Optical pointing test completed in March
2006 - R4.0 - Testing planned for early fall 2006. This
will test the scheduling simulator, particularly
with weather simulation. - TelCal
- R3.1 - Features developed during this release
tested August 21, 2006. Optical pointing
reduction tests completed March 2006. - R4.0 holography reduction tests scheduled late
2006. - Archive
- R3.1 Basic archive function tested during
optical pointing integrated user test in March
2006. - R4.0 Tests of archive queries (projects,
monitor point data) scheduled for integrated user
test late 2006 - Requirements Use Case development for Science
Archive continues. Granular requirements for
Archive starting to be produced. ALMA Science
Archive (ASA) requirements about ready.
5SSR Testing Status (cont.)
- Correlator
- R3.0 - Testing planned for April, June, and July
slipped to October 2006. - Pipeline
- R3.1 - End to end user test (from flagging to
imaging) is planned for the pipeline in Fall
2006. - Quick Look R3.0 - Operations test should be
possible. It is not yet planned. - ObsPrep
- R3.1 User test about to conclude.
- Control
- R3.1 Optical pointing features associated
tested during integrated user test in March 2006. - R4.0 Holography features scheduled to be tested
late 2006 during integrated user test. - Requirements adopting EVLA control requirements
(modified for ALMA) for ALMA control granular
requirements. Passed System Requirements Review. - Offline
- User Test 4 R2.1 - Single baseline data
reduction and analysis - supporting the ATF. Test
complete, report written. - User Focus Group completed March 2006, provided
input on python user interface. - User Test 5 Will test single dish reduction
imaging. Scheduled for Dec 2006 date dependent
on ACA requirements. - Note Control and correlator are exercised not
only at ATF, but also every day in the Prototype
System Integration (PSI) lab.
6Function Based Team (FBT) development
- Works to functionality across subsystems (2
months) - Based on Technical Use Cases
- Interface issues clarified soon
- Allows more frequent integration
- Some parallelism possible
In principle planning of R4.0 FBTs
7Latest Developments
- Concluded R3.1 release and integrated in April
- Includes Optical Pointing e2e functionality
(tested in March at ATF) - Installed and tested on both prototype antennas
- Installed also in the Prototype System
Integration Laboratory in Socorro - Included integrated regression tests
- FBTs so far
- Optical Pointing (completed in March)
- Shared Simulation of Single Field Interferometry
(SFI) (April) - Monitor DB optimization (July)
- On-going FBTs
- ALMA Project Data Model (APDM) re-factoring (Aug)
- Validation almost complete. Currently on hold
behind higher priority Holography work. - Holography (Aug). Hw for tests in the lab early
Sept., at ATF late Sept. - Single Field Interferometry II (to be used at ATF
just after first fringes- achieved with PSI
scripts)
8OPT User Test (mar.06) (1st integrated user
test of ALMA software)
- evaluated by members of the Computing IPT (CIPT)
in March 2006. - primary software limitations were found to be
jlog instability and slowness, lack of coherent
logging levels across subsystems, archive access
(query mechanism not documented at the level of
end users), lack of access to archived monitor
points, missing CCL offset commands, and uncaught
exceptions (e.g. Control, TelCal) that caused
software crashes. - Many of the interfaces and system responses have
been designed for the developer. - Approximately 170 defects, issues, and
enhancements were identified in this report.
Roughly 40 (70) of those are defects (14 (8)
priority 1 and 56 (32) lower priority).
9Next Developments ()
- R4.1 ( up to June 2007)
- Single Dish (SD) Function Based Team (Feb.) (w/o
nutator) - TP mode 1 OTF mapping in continuum Total Power
- TM mode 2 radio pointing measurements using TP
- Test SFI with h/w at ATF (completion of SFI II
work) - Readiness Review (external) by subsystem (May)
- R5.0 ( up to Dec 2007)
- SD Nutator, Spectral line (prepare for equipment
in Chile) - Interferometry WVR, fast switching, Pointed
mosaics - Respond to AIV use of software in Chile
- PAR (Dec.07) Delivery of modes developed thus
far to Chile - During all of 2007 support in Chile for antenna
- Acceptance and integration
- System characterization using continuum TP
- () Development plan will be adapted to
availability of h/w CIPT formal user
test will occur twice next year
10Archive Software
- Persistency support for raw data, pipeline
results (images and logs), SBs and all other User
input, site characteristics, monitor data,
calibration data, catalogs, and all other
auxiliary information required for ALMA to
observe. - Quick look results are temporarily stored in
the Archive. - Items not specific to ALMA Science Archive
- User Repository Manage database of registered
users, support proprietary periods. - User query, retrieval and monitoring capability.
(event data R4.0) - Data replication between OSF, Santiago, ARCs
(both via network and physical media). - Use of NGAS supported data replication. Requires
ALMA testing. - Request Handler Manage packaging, request
tracing and delivery of data products to the
scientific user community (implementation planned
for Q4 2007) - High-availability system with failover (test
implementation Q3 2007) - Provision for safe backup of all data when more
than one Archive is operational this may be
accomplished via data mirroring between Archives. - Support 6 MB/s average and 60 MB/s peak
(sustainable for many hours at a time) data rates.
11Archive Replication between SOC and ARCs Full
description on Archive Twiki at http//almasw.hq.
eso.org/almasw/bin/view/Archive/DatabaseReplicatio
n
- Bulk data replication taken care of by NGAS.
- 2. Transaction based replication for Oracle DB
OSF-SCO - 3. Standard DB incremental backups sent via NGAS.
12ALMA Science Archive (ASA)
- Scientific view on ALMA observations through ALMA
data products (cubes). - Preview image, possibly via an interface to
Science Pipeline. - Science Archive Requirements document was drafted
and has been discussed by the SSR. - Many use cases for the ASA have been drafted and
were given to the developers. (http//almasw.hq.es
o.org/almasw/bin/view/Archive/AsaUseCasesList) - Support for storage and association of advanced
products. - Catalogue Support offer access to catalogues
(calibrators, spectral lines) and calibration
database (R4.0) - Consists of two major parts
- ASA portal Main access point to all query and
retrieval services. - ASA data repositories and ASA services.
- VO aware by sharing technologies and standards.
- ASA VO registry (planned)
- ASA VO SkyNode (planned)
13ALMA Science Archive (ASA)
- Exercise use cases for each user type
- ALMA staff, Archive Users/PI and Reduction
Pipeline - Test interfaces (query structure, ease of use)
transfer content (connections between APDM
structures in ASA made distributed correctly)
performance. - Test data transfer from Operations Archive and
ASA.
Operations Archive (Santiago)
Dataset
ALMA Science Archive
Queries
Query Responses
Commands
Archive Subsystem State
Data
Correlator Data
Processed Data
Quality Data
Archive Users, PI, VO, Staff
ALMA Staff
Reduction Pipeline
14Add a user to the repository
Support for ASA users Staff PIs User
repository development manage database of
registered users
Manager view of users
User edits his/her contact information
15Archive table browser prototype
list of ALMA Science Data Model (ASDM) tables
entities table populated by Correlator (actual
contents not shown)
16Observing Preparation
- Support creation of Phase 1 proposals and Phase 2
programs, with submission through the ALMA
archive. - APDM refactored (R4)
- Visual Spatial Editor first version. Visual
Spectral Editor also exists. Final versions of
both for R6. - Support for Directors time (R5)
- Special SBs for fixed period scheduling now
available. (R4) - Phase I bureaucracy and textual cases (R5).
- Phase I II transition (creation of Phase II
from approved Phase I) (R5) - Support all ALMA Observing modes
- So far optical pointing and holography
observatory modes, and Single Field
Interferometry (SFI) observing mode. - infrastructure in place to allow rapid
development for further modes, R5-6 and on. - Provide a mode which operates in terms of science
goals with automated generation of SBs. - Science goal-based input for Phases I and II
First wizard now available for SFI. Will
expand as modes are developed. - A generic framework for science input collection
and SB generation is in place. - Alternative science goal presentation
(workbooks - more flexible for users than a
wizard) under development R4. - Almost complete infrastructure will allow science
goal input and SB generation to be extended to
more observing modes. R5-6 and on.
17Observing Preparation (cont.)
- Exposure time and data rate calculators shall be
provided. - ETC complete in maintenance. Supports all
arrays. - Data rate estimator first version available,
requires more user friendly presentation (R5.1) - Execution likelihood estimator (R5)
- Sensible default values and basic validity
checking will be provided. - Infrastructure to support validation and setting
of (sensible) defaults is in place. - Needs to be rolled out for all items (R4.1)
- Provide an expert mode with access to technical
parameters of interest to observers (e.g.,
detailed control of correlator configuration). - Almost complete for main array in maintenance
as other specs emerge. Still details to be
determined for ACA/TP arrays. (R5) - Local disk persistency (provided by
MicroArchive). In maintenance. - Source list and catalog (target and spectral)
support - Infrastructure in place with interim catalogs.
- To be moved to Observatory Characteristics (R4)
- Final catalogs in place by agreement with
Operations (R6?) - Widely portable software.
- Supported now, will continue.
18Spatial Visual Editor
19Observatory Operations Software (ObOps)
- Passed its PDR at CDR4 (June 2006). Still at
.5 FTE/yr effort - Infrastructure Ops Support Database, Data Access
Module, etc. (R4-R9) - Shift Log Tool record all day- and night-time
observing activities - core features (R4)
- basic features (R5)
- full implementation (R6)
- Observatory Characteristics Database GUI (R4.1)
- Phase One Manager support for proposal review
process (R6) - ProjectTracker control and monitor operations
information flow (R6) - User Portal integrated access to operations info
- internal (ALMA only) access (R7)
- public access (R9)
- QualityTool data quality assurance (R7)
- DataPacker
- raw data delivery (R7)
- data package production and delivery (R9)
- PublicationsDatabase (R9)
20Offline software
- Priority 1 requirements. In particular for the
next release - Single dish analysis, High dynamic range imaging
(R4.0) - Standard calibration (R4.0)
- CASA framework doc.
- Portable to a range of Unix-like operating
systems. - Efficient enough in operations to satisfy user
expectations and for use with the Science
Pipeline. - GUI ,CLI, and script interfaces (Image viewer
data visualization GUIs available) - Readily installed by end users for processing
execution. GUI tools for raw data and image
inspection, flagging, arbitrary plotting. - Filler for ALMA Export Data Format for
Interferometric data (single dish ASDM filler to
be developed when ASDM format defined). - ASDM extensions, EDF doc (R4.0)
- Extensive support for multi-channel,
multi-polarization, multi-spectral window,
multi-pointing flagging, calibration, imaging,
and analysis for interferometric, single dish,
and combined data from existing telescopes. - Simulation capability.
- Processing history, interactive warnings, and
re-execution facilities. - Direct access to the ALMA Archive and the VO.
- Atmospheric modeling for the ALMA bands
- Extensive user documentation
21CASA
- AIPS
- Glish interface unknown, unsupported outside of
NRAO - Tasking system based on Glish
- GUI system based on Glish/Tk limited widgets,
not robust! - Difficult for external developers to contribute
- Multi-CD binary distribution
- Large monolithic libraries with cross
dependencies - No namespace
- Freeze 2006.75
- CASA
- Python interface (community standard) IPython
extensions - Direct binding to Python ACS, other frameworks
readily possible - Hierarchical set of small libraries with clearly
defined dependencies - Namespace protection for integration with other
code - RPM distribution mechanism auto-updates possible
- Robust
- Inherits all application code improvements in
robustness and performance.
22DataCapture
Maps telescope, observing information into Alma
Science Data Model (ASDM) for post-processing in
near-real time.
23CASA Timeline
- EVLA external science testing wide band
calibration imaging (CASA) - ALMA P1 SSRs complete
- 2008
- EVLA external science testing RFI (CASA)
- NRAO user support (CASA)
- ALMA commissioning support
- EVLA P1 SSRs complete
- 2009-2010
- ALMA/EVLA commissioning support
- Community CASA released/distributed for early
ALMA/EVLA science - 2011
- Community CASA released/distributed full user
support (ALMA/EVLA P1/P2s)
- 2006
- NRAO NAUG testing (AIPS)
- Community Project tests
- ALMA external science testing single-baseline
commissioning test) (CASA) - ALMA external science testing single dish
reduction (CASA) - ALMA commissioning support (CASA)
- ALMA/EVLA user interface review (CASA)
- EVLA external science testing full
polarization imaging antenna pointing
calibration - 2006.5 AIPS frozen
- 2007
- NRAO User support (CASA)
- ALMA commissioning support (CASA)
24Offline User Test 4 (report Apr.06)
- Test delayed by SSR (not developers) so the first
framework changes and python commands could be
evaluated in a realistic environment of single
baseline interferometry. - CASA is a python binding of the C functionality
that was in AIPS. - This initial version of CASA had python commands
that had a one-to-one correspondence with AIPS
Glish commands. - first test of the CASA infrastructure and python
interface. - focus was to exercise and verify functionality
for - editing, visualization, data reduction and
analysis for single baseline interferometry - capabilities that are needed at the ALMA Test
Facility and for commissioning support. - Testers with extensive experience commissioning
radio telescopes and/or operating telescopes. - In general, single-baseline data reduction
procedures worked with good stability and
accuracy. - Additional visualization capabilities to support
commissioning activities still needed.
25Offline Development Test Schedule
Community use, command line python
Limited release, mostly command line, some GUIs
Single dish reduction imaging
Python tasks, new framework
Delay late arrival of ACA requirements.
User Focus group
Tools ? Tasks
Single-BL Commissioning at ATF
1st Python interface, new framework
Delay SSR decision to allow testing of 1st
python interface.
Interferometer/TP combo
Complete
VLAGBT BIMA12m
Small Multi-field Mosaic
VLA BIMA mosaics
User Acceptance Tests
Single Field
VLA PdBI 13mm 1/3mm
2003
2004
2005
2006
2007
26Offline Key Planned Development for 2006/2007
- Commissioning Plan shows heavy use of the offline
package. - Change development priorities/details (e.g.
single baseline analysis software must be ready
earlier than expected more plotting capabilities
needed). - Completed commissioning support test in February
2006 (1st step before using ATF) - Single-dish reduction imaging will be based
on ASAP package - Single-dish user test scheduled for late 2006
(dependent on ACA requirements). - Also evaluate task-based reduction imaging for
interferometric data. - Complete ALMA interferometric ASDM filler and new
viewer (more robust). - Formal tests have been limited to specific
datasets to verify development along a defined
path. This has been necessary for development but
need to expand to more open use once python
command line interface is stable - 2007 2009 Staged plan to release CASA to
astronomers for scientific use. - Not final yet, however, key components are
- Nov 2006 CASA user point person starts (guide
user interface development, documentation help
new users) - 2007-2008 Increase number of NRAO scientists a
few external scientists reduce own data - Mid 2007 Early release of CASA simulator
- Mid-2008 Release to the public.
27Science Pipeline Goals
- Automated flagging, calibration and imaging
with associated quality information for common
ALMA observing modes. Pipeline processing shall
be data driven). - Main purpose is to produce calibrated,
scientifically useful images. - Following information will be calculated or
available for use in pipeline processing
sideband ratios, temperature scale from an
atmospheric model, bandpass calibration,
instrumental polarization, phase rms, antenna
efficiencies from source with known flux, and
time variable amplitude and phase curves.
(heuristics computing bandpass, gain, and flux
calibrations for single field interferometry
data, (R3.1-gt) - Choose whether or not to use WVR corrected if
both are available, to reuse previous
calibrations if necessary, to select appropriate
deconvolution algorithm, to do self-calibration
or not, to do continuum subtraction or not
(Choices non-existent or hardcoded, (R4.1-gt). - Automatically assemble all needed information
from the Archive (i.e., assemble appropriate
sources from multiple SBs over multiple epochs /
configurations). (automatic ASDM retrieval for
observing modes with simple observing unit set
structures supported, (R2.1-gt) - User friendly processing summary and processing
script which can be modified and used in later
reprocessing. (processing summary logs are
written by current pipeline heuristics scripts,
automatic script logging may be available from
CASAPY, (R4.0-gt) - Scripting interface to allow operations to modify
processing heuristics without changes in compiled
code. (Scripts will be written in Python, (R4.0) - Manual mode (e.g., for reprocessing particular
projects). (R5.0-gt) - First version of pipeline processing scripts.
(E2E for single field interferometry (R4.0-gt),
for single dish (R4.1-gt)
28Status Heuristics
- Single field interferometry (prototype e2e
heuristics) - calibrator source baseline based flagging
heuristics - bandpass, gain, flux calibration solution
heuristics - calibrator source imaging heuristics
- some quality assessment measures
- calibrator source antenna based flagging
heuristics - target source flagging and imaging heuristics
- Single dish (baseline fitting heuristics)
- including line detection and masking heuristics
- Software
- general Python classes for flagging,
calibrating, and imaging SFI data - general Python classes for baseline fitting (some
cleanup required)
Heuristics development slower than expected
(based on 2004 planning) Reasons Technical
CASA framework conversion happening at same time
Developmental more difficult to get flagging
heuristics correct than expected Personnel
ramp up to expertise needed to develop
interferometric heuristics slow.
29Status
- Pipeline Infrastructure
- Quicklook
- supports parallel observing sessions (independent
subarrays) - provides clean separation of processing and
display functions - working telescope calibration results monitor and
GUI - Science
- working heuristics development framework
(IPython, CASAPY) - working set of data processing engines (CASA,
ASAP) - heuristics not yet connected to infrastructure
- ASDM filler prototype
- separate ALMA / CASA build environments
- Modules
- full set of tool methods available to heuristics
from CASAPY for - table, ms, constants, quanta, autoflag,
calibrator, and imager - subset of tool methods available to heuristics
from CASAPY for - image, measures
- prototype visibility display tool already
available to heuristics from CASAPY - msplot
30Heuristics User Test 3 Browser Snapshots
PdBI Spectral line
From R3 User test conclusions (mar.06) The
quality of the automatic flagging was judged to
be good by the majority of the testers and the
calibration heuristics were found to be a
reasonable first implementation
31Pipeline - What is ahead for next year ?
- Science
- E2E single field interferometry heuristics
running in the ALMA environment - E2E single dish heuristics
- prototype cluster computing environment
- QuickLook
- user friendly telescope calibration monitor and
display - real time visibility display using real time
filler - some bad data detection heuristics integrated
with alarm system
32Dynamic Scheduling
- Software to automatically choose best SB given
scientific priority, current environmental
conditions, current and future configurations,
and the total pool of possible observations - Complete interactive - and queue - scheduling
(R4.0) (R4.0). - Planning mode simulator (incl. weather data and
quality - R4.0) (R4.0) - SB interruption and re-execution. (Special SBs
for fixed-time scheduling - R4.0) (R5-6) - Manual override of automated choices. (R4.1)
- Mechanisms for operations to change the weight of
factors within the dynamic scheduler without
significant programming. (R4.1) - Scheduler reporting facilities. (GUI for master
scheduler, Exec plug-in - R4.1) (R4.1) - Heuristics delivered by CIPT will require
optimization by operations. (R5) - Multiple sub-arrays supported (including taking
antennas off-,on-line - R4.1) (R4.1 ) - ACA projects operating independently (R5-6)
- Medium term scheduling (R5-6)
- Long term scheduling (R7-8)
- Receiver setup optimizations (R7-9)
- Breakpoint facility low priority. (R7-8)
- Accounting of time spent for various purposes
(source, programs, observatory calibrations).
(R?)
33Operator Master Client
Scheduler
34TelCal Online (and Offline) Calibrations
- Telescope Calibration takes scans (mainly
special calibration scans) and performs data
reduction on them. - Results are used by Control for further
observations with improved parameters and by the
Quick Look Pipeline to display and calibrate
quick look data. - Some array calibrations are not required to be
performed on-line (e.g. Holography, pointing
models). - Holography (Pointing focus offset, illumination
pattern, surface err.) (R4.0) - Atmospheric calibration (Temperature scale) (R4)
- WVR values (R5)
- Sideband ratios (R4)
- Phase calibration (phase rms, calibration curves)
(R4) - Amplitude calibration (calibration curve) (R4)
- Bandbass calibration (R5)
- Reference pointing (publish/reduce scan results)
(done) - Focus (partly done)
- Delay (publish/reduce delay scans) (R4.0)
- Pointing Model (optical done)
- Baseline measurement (positions from delays)
(R4.0) - Skydip
- Interferometry solvers (done)
- Automatic activation (according to intent) (R4.0)