PanSTARRS DATA FLOWS

1
PanSTARRS DATA FLOWS

• Richards 2/17 email
• Need to capture housekeeping I/F e.g., modes

• March 24, 2003

2
Top Level

• Phases

3
Top Level Data Flow and Principle Interfaces
Chip Level
Chip Level
Telescope and System Level
System Level
System Level
System Level
Image Capture
Phase 0
Phase 4 Augmented Image Processing
Phase 5 Science Client Product Generation
Phase 6 Science Client Interfaces
Phase 1 Detector Calibration (Calibration and
Instrument Correction Processes)
Phase 2 Map and Warp to Sky (Image Manipulation
Processes)
Phase 3 Create Sky Image (Image
Combination Processes)
Science Clients
Scheduler
Data Storage
Data Storage
Data Storage
Data Storage
Data Storage
Data Storage
TCS and Environment Monitoring
Reference Catalog Generation/ Update
Fringe Image Generation/ Update
Calibration Functions (Not complete)
Mission Planningpre-staging of each nights
scheduling and supporting dataTBD
4
Second Level

• Data Flows and Internal Interfaces

5
Phase 0 Image Capture
Index image is responsible for assigning a unique
ID to each frame file off the scope, and
associated metadata record i.e, we modify the
metadata record and append/pre-pend to the
digital data file that goes on to the raw data
buffer.
0.2 Assign Image ID
0.1 CCD Control/ Data Processing
Mission Planning Function loads up the guide
stars data store. Note that the guide star
catalog may actually specify a region of space to
search for a guide star to compensate for
telescope pointing area. It might be better to
implement an algorithm that searches the "track
area" of the CCD for appropriate stars from the
stars seen by that area. The guide star
referenced here is not to compensate for
telescope pointing error but rather to provide a
reference so that atmospheric turbulence induced
image wander may be tracked out. As such, the
requirements on these stars are fairly loose,
basically non- saturating and a good S/N ratio.
Telescope/ Camera
Phase 1
Digital data mirror image
TCS
Scheduler
Metadata mirror image
Image metadata store will (eventually) contain
mission support data, including bias, light
histogram, pointing info, observed sky, local
weather, and TBD scheduling information. Starting
with Phase 1, there is a mirror image of this
data store retained at MHPCC distinct IDs
6
Phase 1 Detector Calibration
Assumes only digital data cross the interface.
Starting with Phase 1, these sets of mirror
images are assumed to be at MHPCC distinct IDs
Do we replace CCDs? If so,then we need
re-calibration data, which are updated
periodically as detectors age
1.2 Separate Data and Bias Info
Phase 0
We have the option here of setting
defective detector cells to the average noise
level (hence noise input or in defining large
negative numbers as flags and using them to flag
defects. Combination strategy will drive choice.
We can assume Markovian, which means we need only
last nights data.
1.4 Remove DetectorDefects
1.3 Remove Bias
bias
Bias Store
DS1.2-1 Diagnostic (TBD)
1.7 Detect Sky and/or Scattered Light
1.8 Remove Sky and/or Scattered Light
Phase 2
1.5 Divide by Flat Field
1.6 Subtract Fringing
Fringe Image
Super Flat
Histogram Catalog
DS1.5-1
DS1.6-1
Fringe effects can change during night, may need
frequent (daily?) updates from mission planning
function
DS1.8-1 Diagnostic (TBD)
7
Phase 1 Internal Product Generation
(Calibration)Fringe Image Update (1.10-1.19)
Phase 2
1.10 Combine "N" Previous Sky Images
Phase 1
Note A quality check of this operation would
take a Fourier Transform of the current image
(to measure periodic structure). This periodic
structure would be compared with that measured
for the updated fringe image. If there was more
periodic structure in the updated fringe image
than in the current image, fringe reduction would
really be fringe enhancement. An alert then
should be issued and step 1.6 turned off until
reset with next image.
Interface 1
Interface 6
NOTE These processes may not be conducted for
each observation but may be triggered when
appropriate
8
Phase 2 Map and Warp to Sky
How many references in each image are required to
ID? This is updated/ replaced with the contents
of 5.57 periodically (2 yrs stated at CoDR?)
2.3 Generate Spatially Variable PSF
2.5 Determine Warp and Border Boxes
Phase 1
2.4 Perform Object Cross Identification
2.2 Flag Cosmic Rays
2.1 Find Objects
Note Dave Monet feels very strongly that this
data should be saved both for his analysis and as
a check for data integrity/testing through the
processing. He would try to fund this part of
the storage if needed. This could result in a
very big file over time (the big file would be
the archive in phase 6. This file would dump to
that one periodically and be cleared.) ??Keep
objects for each scope, or a reference scope?
(Stationary Candles)
2.10 Resample on Sky Coordinate Grid
2.6 Determine Photometric Scaling
2.9 Perform Frame Warping
2.8 Map to Sky Coordinate Grid
2.7 Determine Noise for Each Cell
Phase 3
Telescope-centric coordinates, processes
performed for each Telescope
9
Phase 3a Create Sky Image(done on a
per-telescope basis)
Note Function 3.8 also may incorporate logic
that would discriminate against pixels very near
the edge of a frame in preference to those near
the center. This would save storage relative to
a scheme that saved overlapping plane images.
Assumption The telescopes are very close in bore
sight angles and hence have most gaps in common
so will have to do multiple exposures to capture
blanked-out areas (cadence of 20-50?)
Created over a weeks scanning to capture the
weeks sky holds pixels or flags indicating CCD
gap/defect/cosmic ray/low orbiting object. With
effective nutation/overscan no flags should be
left after whole week's data accumulated
3.9 Perform Noise- Weighted Image Update
Phase 6
"normal" image
Phase 2
Accumulated Sky Store
3.7 Assemble Noise Map
warped images
3.5 Route Normal/ NEO Images
Phase 3b
metadata
3.6 Assemble Telescope Gap Map
metadata
3.8 Create Image Exposure Map
Phase 5 Full color sky
3.10 Route Reference/ Velocity Images
NEO images
Phase 3b
Note the reference and velocity image are
artifacs created so that the same night NEO visit
and revisit are not combined in the image store
but buffered by themselves to allow extraction of
velocity vector info
10
Phase 3b Combine Sky Images (done on a system
basis)
entire sky
3.2a Detect and Blank Cosmic Rays (CRs)
3.4a Perform Noise- Weighted Image Combination

Phase 4
Phase 3a
DS1.1-2
3.1 Merge Image Metadata
Image Metadata
Phase 5 Phase 6
3.2b Detect and Blank CRs
3.4b Perform Noise- Weighted Image Combination
Phase 4
sweet spots only
3.4c Perform Noise- Weighted Image Combination
3.2c Detect and Blank CRs
11
Phase 4a Augmented Image Processing
Phase 4b
4.1a Create Difference Image
Sky Image
Difference Image
Phase 3b
Phase 5 (NEO)
4.1b Create Difference Image
Reference Image
Reference Image
Velocity Image
Velocity Image
for NEO output
Updated in 4b
Interface 5
4.1b Create Difference Image
Interface 4
12
Phase 4b Augmented Image Processing
Note The short-term cumulative sky image is
limited in its time span to prevent
motion-smearing of near-by stars. The Cumulative
Sky Image has a longer history as its focus is on
enhancing the seeing of galaxies and smearing of
near-by objects is not a problem in the study of
distant galaxies
How often is I/O to sky image requested? Are
files neighboring? Will multiple files need to be
retrieved to cover difference image e.g., to
cover the warped corners? This should not be a
problem if a consistent "plate" size and grid
spacing is used to organize the data.
Exposure Map
Phase 3
Sky Image
Phase 6
4.2 Subtract Difference Image from Sky Image
4.3 Update Cumulative Sky Images
Phase 4a
Difference Image
Phase 5
Note it is the intention of this step to remove
the variables
This is a sliding window cumulative image with
old contributions dropped off to prevent motion
from blurring the near star images.
Interface 5
13
Phase 5A Science Client Product GenerationBasic
NEO Identification (5.1-5.9)
TBD More detail needed to specify algorithm for
QC, which depends on how many we expecttoo many
for real people to process
Difference Image
Phase 4
5.7 ID New Object
new
5.2a Remove Variable Objects
5.1a Detect Objects
Phase 6
5.4 Link Orbits
5.8 QC
5.3 Calculate Velocity Vectors
known
known
unknown
5.6 Prepare Knowns Update
5.5 Prepare Unknowns Update
Updated in 5D
5.1b Detect Objects
5.2b Remove Variable Objects
Velocity Difference Image
Interface 5
NOTE These processes may not be conducted for
each observation but may be triggered when
appropriate
14
Phase 5B Science Client Product Generation
Master Catalog (5.10-5-14)
Something that help with the distinct modes
e.g., red vs blue filters may be same as
analogous stores in 5C and 5D
Phase 4
Cumulative Sky Image
5.14 Prepare Object Catalog Updates
5.10 Find objects
5.11 Classify Objects
5.12 Photometry
5.13 Astrometry
Phase 6
???TBD Do we need a bright/planet suppression
processing thread?? The stationary candles and
other reference catalogs may be created in the
"calibration" pipeline which will be a process
differing from this one in that the very bright,
very dim, and moving objects will be deleted to
produce a "thinned" catalog that will provide the
photo metric and position references needed while
minimizing the computational loads. The upper
and lower clip levels will be a function of where
in the sky one is observing (some places have
more stars than others)c
Catalog Updates
Updated master catalog is copied for publication
as internal catalog in Phase 6.
Interface 4
NOTE These processes may not be conducted for
each observation but may be triggered when
appropriate
15
Phase 5C Science Client Product Generation KBO
Observation (5.20-5.29)
Note This may become the same process as shown
in Phase 5A, especially if the photometry and
astrometry functions are needed in 5A
Something that help with the distinct modes
e.g., red vs blue filters may be same as
analogous stores in 5B and 5D
Interface 6
Difference Image
5.21 Classify Objects
5.22 Photometry
5.23 Astrometry
5.20 Detect Sources
Phase 4
Phase 6
Interface 5
5.29a Update Observational History
known
5.25 Link Orbits
new
unknown
5.26 Prepare Unknowns Update
5.27 ID New Object
5.24 Calculate position
5.28 QC
5.29b Prepare KBO Update
NOTE These processes may not be conducted for
each observation but may be triggered when
appropriate
16
Phase 5D Science Client Product Generation
Variable Stationary Objects (5.30-5.39)
Something that help with the distinct modes
e.g., red vs blue filters may be same as
analogous stores in 5B and 5C
Interface 6
Phase 3
5.33 Cull Moving Objects
5.30 Detect Sources
5.31 Classify Objects
5.32 Astrometry
5.34 Photometry
Combined Sky Image
Phase 6
Example of transient data store required for
tuning of algorithms and/or recovery more to
come
5.36 Prepare Variable Object Catalog Updates
May only occur periodically
5.35 Update Observation History
May only occur periodically
Catalog Updates
Interface 5
17
Phase 5E Science Client Product Generation Full
Color Sky Image (5.40-5.49)
Something that help with the distinct modes
e.g., red vs blue filters not the same as in
5B-E one for each scope
5.42 Photometry
5.41 Detect Objects
Phase 3 Create Sky Image Branch a
5.42a Photometry
5.43 Compute Pseudo- spectrum
5.41a Detect Objects
a
Phase 6
5.42b Photometry
5.41b Detect Objects
b
Phase 3 Create Sky Image Branch b
c
5.41c Detect Objects
5.42c Photometry
Phase 3 Create Sky Image Branch c
5.42d Photometry
5.41d Detect Objects
d
Phase3 Create Sky Image Branch d
5.44 Combine images
Note, these outputs are sent in separately, not
combined
Interface 5
Interface 6
NOTE These processes may not be conducted for
each observation but may be triggered when
appropriate
18
Phase 5F Internal Product Generation Reference
Catalog Update/Calibration (5.50-5.59)
Need a separate version of stationary candles to
update must keep stationary candles
un-modifiable for Phase 2. But, do we
periodically update/upgrade stationary candles
for use in Phase 2? Also use the updated catalog
to replace contents of guide stars in Phase 0.
Ans Update ref sky 2 years, and do this with
H/W upgrade, but keep all versions to re-process
images later. NOTE other data must be stored as
well for re-processing TBD mission support data
Interface5.1c
Interface5.2c
Phase 4
Cumulative Image
5.52 Find Objects
5.53 Classify Objects
5.50 Remove Bright Sources
5.51 Remove Dim Sources
5.54 Delete Extended Objects
Phase 0 Phase 2
5.57 Prepare Reference Catalog Updates
DS5.57-1
Updated Reference Catalog
5.55 Photometry
5.56 Astrometry
Interface 4
Something that help with the distinct modes
e.g., red vs blue filters may be same as
analogous stores in 5B5E
NOTE These processes may not be conducted for
each observation but may be triggered when
appropriate
19
Phase 6 Distribution and Data-Based Scientific
Product CreationData Flow and Internal Interfaces
Published archives are copies of corresponding
(updated) internal stores, buffers, catalogs, maps
Phase 5
NEO Server
Phase 6
Preferred Science Clients
Stars and galaxies only??
National Virtual Observatory
6.3 Internal search and presentation engine
6.1 Publishing process
6.2 External search and presentation engine
What can PSC/NVO request to be retrieved? How
often? How does metadata support the search? See
SDSS site for example of this function. Metadata
not supplied to these users
Combined, cumulative, and full color images
their metadata
Interface 7
Interface 6
Image its metadata Database without bounds!!!
20
Third Level

• Objects, Macros, Custom Code Etc.

21
Process x.y, Qwaetasytst
PROCESS
OUTPUT
INPUT
Parameters
Items To Data Store
Items From Data Store
DATA ASSURANCE TESTS