Science Data Systems SDS

1
Science Data Systems (SDS)

• CUC Presentation
• Jan 2005

2
Science Data Systems - supporting user science
with software and data products

• Work closely with Data Systems to provide science
  overview of software and detailed specs
• CIAO - the software package (and documentation!)
  for user data analysis
• Pipelines - ensure data products created with
  correct algortihms and contain the needed
  metadata
• CALDB - figure out how raw calibration results
  from CAL team must be packaged to support user
  analysis and data processing
• From CUC point of view, we mediate between users
  and Data Systems teams to deliver improved user
  software

3
Science Data Systems - under new management

• Jonathan McDowell, SAO - group leader
  jcm_at_cfa.harvard.edu
• Mike Wise, MIT - deputy leader
  wise_at_space.mit.edu
• SDS is about 50/50 SAO and MIT an integrated
  team
• Close coordination with Pepi Fabbiano's Data
  Systems groups, especially Janet DePonte Evans'
  software development teams CIAO and the
  pipelines are joint SDS/DS products
• Let us know how we can improve your ability to do
  science with CIAO and with CXC data products

4
Science Data Systems summary

• Recent releases
• CIAO 3.2
• APPHOT and DS9 interaction
• Pileup Report
• Documentation
• Estimation
• Correction
• What else is coming
• Improvements
• Enhancements

5
CIAO 3.2 Highlights

• New 'How does the new release affect your
  analysis' web page
• New mkacisrmf tool released
• Spatial model of the ACIS contamination
• New ACIS hot pixel tools
• CALDB 3.0

6
CIAO 3.2 Web Pages

7
PHA

?
mkacisrmf infileacisD2000-01-29p2_respN0001.fits
outfilesource_pi.wrmf \
energy0.110.00.05 channel110241 chantypepi
\ wmapspectrum_pi.fits gainacisD2000-01-2
9p2_gain_ctiN0001.fits
8
Released mkrmf RMF
New mkacisrmf RMF
9
Spatial Model for the ACIS Contaminant
Temperature structure of ACIS OBF
The ACIS QE degradation model has been enhanced
to account for spatial variations in
the contamination on the ACIS optical blocking
filters. The contamination is now expressed as a
function of time, energy, and ACIS chip
coordinates.
10
Spatial model implemented in all CIAO 3.2 tools
11
Status of APPHOT Package

• Frank Primini's X-ray aperture photometry scripts
  demoed at last CUC as an example of scripting
• You asked for more details on their availability
• They are now real software and users can have
  them!
• And you asked how we did those cool things with
  ds9
• I'll tell you how, and we'll soon put this up as
  a thread for users
• Working with Data Systems to formalize how we
  develop and release interactive scripts as we
  increase their use. Stay tuned.

12
Status of APPHOT Package

• Version 1.1.0 available for interested users
  athttp//hea-www.harvard.edu/fap/APPHOT/. Will
  be added to CIAO contributed software after
  additional testing.
• Installs under CIAO 3.2 no additional
  installation of XPA or SLxpa modules required
• Can now determine actual ECF of aperture defined
  by user
• Faster computation of Theta, Phi (now uses S-Lang
  pixlib module)
• Version 1.2.0 in development.
• Choice of method for accessing ECF data file
• Interpolation in theta, phi, energy
• Nearest theta, phi, energy in ECF data cube
• Plots of ECF vs. aperture radius

13
Status of APPHOT Package

• Additional development suspended pending
  evaluation of Penn State's ACIS Extract S/W
  package
• ACIS Extract richer, more sophisticated. We are
  considering porting its IDL-based functionality
  to CIAO's open source environment
• APPHOT could still be used to provide quick
  assessment of aperture sizes, e.g., for proposal
  planning

14
Launching CIAO Tools from ds9
The ds9 image display program can be used to
launch user-defined analysis tasks which can
obtain input parameters from and display results
to ds9. This capability is accomplished by means
of ASCII analysis commands files, loaded into
ds9 via the Analysis Menu, and is described in
detail in the ds9 Reference Manual. Typically,
this makes sense when one is examining an image
and wishes to define regions or image locations
at which, say, counts or PI spectrum are desired.
These can be obtained and displayed quickly using
CIAO tools. In the following pages we present
some examples of analysis command files that
launch CIAO tools.
15
A Simple Analysis Commands File

• The analysis commands file (ACF) is a structured
  ASCII file that contains command lines plus
  keywords that tell ds9 how and when to run the
  command. The command file can take advantage of a
  number of ds9 internal variables, or macros, that
  expand to arguments containing information on the
  current ds9 display, prior to command execution.
  The full list of macros is provided in the ds9
  Reference Manual. Of particular importance are
  the following
• filename expands to the filename of the
  displayed image or event list
• regions expands to the list of regions (in
  various formats) defined in the display
• text displays any text output from the command
  to a text window.
• The ACF can also define parameters, which are set
  in a parameter window, and which may be passed as
  arguments to the command line.
• The following page contains a simple ACF that can
  be used to display some ds9 macros and run the
  CIAO tool dmlist to display header or data.
  Results are displayed on subsequent pages.

16
The ds9_ciao.ans ACF
17
Loading the ACF
Clicking the Analysis button allows users to load
ACFs via the Load Analysis Commands submenu
18
Running dmlist
Clicking dmlist invokes the parameter window
Clicking on dmlist invokes the parameter window,
in which values of the option parameter may be
set
Clicking OK runs the dmlist command line
19
Results for dmlist
Clicking dmlist invokes the parameter window
Clicking on dmlist invokes the parameter window,
in which values of the option parameter may be
set
When dmlist finishes, the results are displayed
in a pop-up text window
Clicking OK runs the dmlist command line
20
A More Complicated (but still simple) Example
It may be difficult to build complicated CIAO
command lines that can be successfully parsed by
ds9's command line parser. Rather, the user may
wish to pass the arguments obtained from
parameter windows or ds9 macros to a shell script
with sufficient string handling capabilities to
build and execute a valid CIAO command line. The
following page illustrates how that can be done
in a script that runs the CIAO tool dmcoords to
determine the off-axis angle of a point selected
in a ds9 display.
21
Launching dmcoords from ds9_ciao.ans
22
Running dmcoords from a Script
23
Results for dmcoords
Clicking dmlist invokes the parameter window
Clicking dmcoords runs the program to determine
off-axis angle for the image location specified
by this region. Results are displayed in the
pop-up dmcoords window.
Clicking on dmlist invokes the parameter window,
in which values of the option parameter may be
set
When dmlist finishes, the results are displayed
in a pop-up text window
Clicking OK runs the dmlist command line
24
Advanced Use
The previous examples were only intended to
illustrate the basics of setting up an ACF,
defining parameters, and passing arguments to
CIAO tools. In practice, the most flexible way to
launch CIAO tools from ds9 is via scripts that
communicate directly with ds9 via the XPA
Messaging System. This can be accomplished either
by running the various XPA programs (xpaget,
xpaset, etc.) from the command line within the
script, and then parsing the results for the
desired input, or by using a scripting language,
such as S-Lang, which provides bindings to the
XPA library. Users who wish to use these
advanced features should examine the scripts in
the APPHOT package.
25
SDS Efforts for Pileup

• Helping users diagnose and deal with ACIS pileup
  is an SDS top priority for 2005
• I'll present the current status

26
SDS Efforts for Pileup

• Comprehensive ABC Guide (status draft in work)
• One Stop Shopping for Information and Advice
• New Diagnostic Tool For Detecting Piled Sources
  (research)
• Concept Detects Piled PSFs
• Potentially can Help Calibrate MARXs Pileup
  Module
• Tools for Dealing with Piled Gratings Spectra
• Simple Model Soon Available
• Sophisticated Model in Development
• Future Work
• Models of Piled Lightcurves

27
The Chandra ABC Guide to Pileup Outline

• Introduction
• Basic Definitions, Pileup Fractions and Their
  Effects, Methods for Avoiding Pileup, When Not to
  Avoid Pileup
• Estimation
• PIMMS, Simulation with Spectral Fitting Packages
  (Sherpa, ISIS, XSPEC), MARX
• Detection
• Simple Methods, Future New Detection Tool
• Mitigation
• Spectral Fitting Packages (Sherpa, ISIS, XSPEC),
  Simple Gratings Models, (Future) Advanced
  Gratings Models

28
The Chandra ABC Guide to Pileup Goals

• Consolidate Information
• Proposers Guide, PIMMS Ahelp, Sherpa Ahelp,
  Threads,Journal Articles, etc.
• Provide Practical Advice
• Pros Cons of Avoidance Strategies (Greater
  Detail than Proposers Guide)
• Subtleties of Detection Methods
• Subtleties of Spectral Fitting Mitigation
  Techniques

29
New Detection Tool in Research/Development

• Compares PSF from 2-pixel split events to other
  events
• Provides Statistical Assessment of Whether the
  Two Distributions are Consistent (i.e., Not
  Piled)
• Does not Directly Rely Upon Spectra
• Potentially sensitive to Even Mild Pileup
• Side Benefit Statistics Module for CIAO (e.g.,
  K-S Test, Mann-Whitney Test, etc.)

30
PSFs Distort Under Pileup (In Complex Ways!)
Working Concept Split Events PSF Distorts
Differently than that for Other Events
31
Low (Unpiled) Count Rate PSF from Split Events
and Other Grades Appear the Same
(Off Axis Source)
p-value 0.05 Indicates Normalized PSF
Distributions are Indistinguishable
32
Example Radio Galaxy Nuclear Emission,Difficult
to Assess Pileup via Spectra
Observation of Gambrill et al. 2003, AA,
401 Source was Thought to be Unpiled
33
Normalized Grade Selected PSFs Differ
p-value Statistically Significant Good Indication of
Mild Pileup?
34
PSF Can be Assessed with the MARX Pileup Module
Consistent with Mild Pileup
35
Spectral Fit Function, Based on Observed
Counts/Frame/Pixel, for Simple Correction
10 Pileup Correction
Simple Spectral Model, Only Applicable to Sources
with Peak Pileup Approximately Applied to an X-ray Binary Spectrum)
36
More Sophisticated Correction in Development
Will Incorporate Detector Efficiency Variations,
Deadtime Effects, Energy Dependence of PSF/LSF,
etc.
37
Spectra, with Peak Pileup of 20, Corrected
38
Ongoing Work
Requirements for Gratings Pileup Correction

• Better Afterglow Correction acis_run_hotpix
• The Most Recent Gratings LSF mkgrmf
• In Cooperation with Calibration, Need to
  Incorporate Better Models of
• Cosmic Ray Dead Area Correction
• Backside/Frontside Correction
• Iridium Edge Correction
• Exploring Possibility of Creating a Chandra
  Lightcurve Simulation, which Would Include Pileup
  Effects
• Adapt Existing XEUS Lightcurve Simulator for
  Chandra Use
• MARX Provides Accurate Mean (White Noise) Piled
  Lightcurve, this Could Provide Piled Lightcurves
  with Realistic (Red Noise, QPO) Variability
  Features

Future Pileup Work
39
Whats coming in CIAO 4.0

• Infrastructure improvements must keep up with
  users' environment
• Increased platform support (under review - newer
  Linux builds)
• 64 bit and large file support
• Performance improvements (pixlib 10 times
  faster)
• Ability to build from source
• Enhancements filling gaps in user science
  capabilities
• New CHIPS plotting package Sherpa improvements
• Improved Backgrounds (ABC guide, scripts, tools)
• Multi-observation support (mosaicing)
• Schedule and other enhancements still under
  review

40
Sherpa status

• Already in CIAO 3.2 bug fixes for improved
  stability, projection error method improvements,
  and revised manual
• We are now conducting a major review
• Reworking the optimization methods (including
  evaluating adding the LMDIF algorithm) for faster
  and more accurate convergence
• Expect to rework Sherpa to modularize it - make
  separate libraries available for models,
  optimization methods, etc. with defined
  interfaces to let us write small tools and
  scripts using parts of the Sherpa functionality.
  Will also greatly improve Sherpa code
  maintainability and stability
• Must add support for new CHIPS plotting