Enterprise CRM Solution

1
Observing Process Astronomer's Integrated
Desktop Scheduling Block Based Observing GBT
e2e Software Review May 3, 2005 Amy Shelton
(ashelton_at_nrao.edu) Nicole Radziwill
(nradziwi_at_nrao.edu)
2
Ease of Use Project Description from August 2004
SB Executor
Quick Look Data Display
GBT OT
Status Screen
Configuration API
uses
Observing API
BUILD
Balancing API
EDIT
Console Windows
RUN
MONITOR
Easy Access to Documentation, Help
Data Capture
Astronomers Integrated Desktop (Astrid)
Scheduling Blocks Execution Log File
Annotated Index of Observation
Observation Management Database (e2e Project
Database)
Note that APIs are very telescope-specific, yet
are used to abstract the observation into terms
not specific to the telescope
3
GBT Observing Process

• Why do we want an Integrated Desktop?
• Astronomers dont have to learn multiple
  applications to accomplish the observing task
• Remote observers only launch one application,
  manages on-screen real estate well
• Error reports do not require knowledge of what
  application error was reported in
• Why do we want to transition to a Scheduling
  Block based system?
• Observing Motivations
• Enable observation data mining we can track
  what was observed more effectively
• Encourage up front preparation maximize
  throughput of science per observing session
• Enable automated dynamic scheduling
• Facilitate proposal review process we can use
  data from past proposals to review current
• Balance interactivity needs with need to optimize
  high frequency weather
• Simplify routine operations such as regression
  tests and all sky pointing
• Characterize observations better so that pipeline
  processing can be enabled in the future
• Technical Motivations
• Enable more efficient troubleshooting we can
  track what people did, and what errors occurred,
  much more easily
• Code is written to accommodate a discrete number
  of well-defined levels of abstraction
• Distinct categories of usage (astronomers use
  apps and application components, experts use
  HLAPIs, programmers use LLAPIs)

4
Observation Process Preparation Execution
Currently, Observation Process focuses on
single SB execution. The Science Program level
is currently unimplemented.
Adopted on the GBT as defined by ALMAbut
slightly customized for the GBT (e.g.
details of ObsProcedure).
NRAO e2e terminology used throughout
presentation.
5
Observing Process Data Model

• Ties observation to proposal
• Allows data mining on GBT observations
• Schema relatively unchanged from last year
• ObservedIndex unimplemented
• ObsTarget unimplemented
• Security added

Customizedfor usewith GBT
6
High-Level Architecture
GUIs
Application
Application Component
0..n
uses
HLAPIs
Expert User
Programmer
uses
uses
LLAPIs
Control System
Observed Data (in an EDF)
Real-Time Data (streaming)
7
Astronomers Integrated Desktop (Astrid)

• What is Astrid?
• Astrid is a container from which you use various
  GBT applications, e.g. Observation Management and
  Data Display.
• Multiple application instances are supported,
  e.g. two Data Displays.
• Why is Astrid so important?
• Simplifies observation startup. Users simply
  type astrid at any Linux prompt.
• Reduces the number of applications that observers
  have to launch to begin observing to ONE most
  useful for observing remotely!
• Observers do not have to know the difference
  between those applications to report issues.

8
Astrid Architecture

• Application Component Tabs Allows the user to
  launch multiple applications within a single
  container. They provide a GUI interface to the
  HLAPIs for all users.
• HLAPIs are always available to the expert user
  to bridge the gap between non-interactive SB
  based observing and the desired for interactive
  observing. Useful for special purpose tasks
  (balancing mid-scan) and commissioning.

9
Astrid Screen Shot
Drop-Down Menus
Tool Bar
Application Components
Python Command Console Expert User access to
HLAPIs, e.g. Balancing and Configuration
Application Component Log Window
10
Available Astrid Application Components

• Astrid Launches a separate Astrid session
• DEAP Data Extraction Analysis Program
  provides general data display and manipulation
  capabilities.
• Python Editor A text editor that provides
  Python syntax highlighting.
• Text Editor A text editor for general use.
• Data Display Provides real time display of
  data plus offline data perusal.
• Logview Used to examine engineering FITS
  files.
• Observation Management Edit/Submit/Run
  Scheduling Blocks on the telescope.
• GBTStatus (In development) Telescope status
  information.

11
Astrid User Help
12
Observing with Astrid

• Preparation
• Write your Scheduling Block(s).
• Observation Management provides an editor
  environment with syntax highlighting
• Use your favorite editor (Observation Management
  provides an import utility)
• Validate your Scheduling Block
• This is automatically done when importing
  Scheduling Blocks into the Observation Management
  editor and when saving Scheduling Blocks to the
  database
• Upload your Scheduling Block to the Observation
  Management database
• Observing
• Retrieve your saved Scheduling Block from the
  Observation Management database
• Submit the Scheduling Block to the Job Queue
• Promote your Scheduling Block from the Job Queue
  to the Run Queue
• Monitor Progress
• Observation Management Monitor tab
• Monitor telescope status with GBTStatus
  application component

13
Observing with Astrid
In the unified environment of Astrid, you can
edit SB, submit SB, monitor SB progress, check
GBT status, and view Continuum data with the Data
Display.
14
Observing with Astrid
In the unified environment of Astrid, you can
edit SB, submit SB, monitor SB progress, check
GBT status, and view Continuum data with the Data
Display.
15
Scan Types Observing Directives
Observing API
Scan Types
Slew Track OffOn OffOnSameHA OnOff OnOffSameHA
Tip DecLatMap DecLatMapWithReference
PointMap PointMapWithReference RALongMap
AutoFocus AutoPeak AutoPeakFocus Focus Nod Peak
Observing Directives
Python style comments, Annotation, Comment,
Configure, Balance,
Break, execfile, DefineScan, SetSourceVelocity,
SetValues, GetValue
Documentation available on the GB Wiki at
Observing.ScanTypes Software.ObservingDirectives
16
AutoFocus, AutoPeak AutoPeakFocus

• Syntax (note all parameters are optional)
• AutoFocus(source, frequency, flux, radius)
• AutoPeak(source, frequency, flux, radius)
• AutoPeakFocus(source, frequency, flux, radius)
• How does it work?
• Finds current beam location and receiver
• Configures using standard continuum configuration
• Selects nearby calibrator (uses measured sky
  system temperature for Peaks)
• Balances
• Sets source name (comes from Jim Condons catalog
  uses J2000 syntax)
• Peaks/Focuses
• Parameter Info
• source is a string and specifies the name of a
  particular source in the pointing catalog to be
  used for calibration. The default is None.
• frequency is a float and specifies the observing
  frequency in MHz. The default is the rest
  frequency used by the standard continuum
  configuration cases.
• flux is a float and specifies the minimum
  acceptable calibration flux in Jy at the
  observing frequency. The default is 20 times the
  continuum point-source sensitivity.
• radius is a float. The routine selects the
  closest calibrator within the radius (in degrees)
  having the minimum acceptable flux. The default
  is 10 degrees.

17
Lessons Learned Deploying Software on Working
Telescope

• Technology Transfer
• Astrid deployment delayed because technology
  transfer between Software Development staff and
  Observing Support staff is taking much longer
  than originally planned for
• Staged Deployment
• Program Layer not yet implemented
• Observers expectations already influenced by
  previous observing experiences at the GBT
• Abuse at SB layer
• Long SBs
• for loops
• Training documentation provided to encourage
  best practices

18
Lessons Learned Deploying Software on Working
Telescope

• Up-front preparations
• Major paradigm shift, especially for support
  staff
• Observing strategies can be developed minutes
  before observing, or even on the fly, but
  planning minimizes user error and lost telescope
  time
• Constructing SBs requires forethought
• Authorized projects must be entered into database
  ahead of time
• Observer name must be entered into database ahead
  of time, and associated with their projects
• SBs need to be validated for observational
  integrity
• Ultimately will make most efficient use of
  telescope time
• Dedicated Forum for Software Development Response
• Captures valuable user feedback
• Visible indicator to Support Staff of Software
  Developments response to user-initiated
  suggestions/issues.

19
Lessons Learned Aligning Development with
Organizational Goals

• Remote observing needs require simplified
  application startup
• Make most efficient use of bandwidth
• Reduce number of applications needed to observe
• One of the major issues driving the development
  of Astrid
• Early tests using VNC very promising
• Importance of validation a priori
• To reduce lost telescope time, SB must be created
  before scheduled observation.
• Validation currently catches syntactic errors
  ahead of time
• Plans to expand Validator to further reduce time
  lost to non-syntactic user error, e.g. passing an
  illegal file to Configure
• Interactive observing is still a requirement
• Support commissioning activities, e.g. new
  receivers
• Support pulsar observations, e.g. balancing
  outside a SB
• Astrids Python console provides access to HLAPIs
  that supplement observing intent captured in SBs.
  Provides on the fly access to the telescope.

20
Lessons Learned Additional Requirements

• Project Data Directory
• Missing mechanism for specifying project data
  directory
• One project, many sessions
• Sessions kept in separate directories
• Sessions may have overlapping Subscan numbers
• Need for Online/Online (monitor only)/Offline
  Modes
• One user should have control of telescope at a
  time
• One or more users would like to see what is going
  on, e.g. support staff, operators, collaborators
• Operators have ultimate control
• Users with upcoming observations need mechanism
  for submitting SB to database
• Additional Control Mechanisms
• E.g. device manager on/off
• How much direct interaction with the control
  system should be enabled in the technology?
• Usability Issues and Bugs
• Accumulating user feedback, which will be
  prioritized and implemented as resource
  allocations permit

21
Enabling Full Functionality in 2005

• Described at http//wiki.gb.nrao.edu/bin/view/Soft
  ware/ObservingToolsv2
• Institute Online/Offline Modes C3
• Provides appropriate level of access to users at
  each stage of observing Preparation, Execution,
  Data Reduction
• Improve Responses to Stops, Aborts Control
  System Failures C4
• Moving Sources Source Catalogs C5, C6
• Eliminate requirement for users to set source
  names and velocities.
• Eliminate need for users to create own source
  catalogs unless desired.
• Improved SB Submission FIFO Queueing Model
  C6, C7
• SB observing with Astrid requires much mouse
  clicking
• SB Job/Run Queue modifications to support batch
  or interactive SB submission
• Provide SB management capabilities to users
• Enable Offline Validation of SBs using
  full-telescope software simulator bound to
  production control system software C8

22
Remaining Issues

• 1. Implementing a GUI to build Scheduling Blocks
• Observing Issues
• Bla Bla
• Technical Issues
• We have a Java prototype build last year that
  looks very similar to ALMA OT, but builds single
  SBs and does not manage Science Programs
• Should we
• Should it reverse engineer SBs to GUI settings?
• 2. Enabling Science Program Management

23
Conclusions

• Because of cooperative work between SDD and
  scientists this year, we can expect complete
  transition of GBT observing for all standard
  observing modes to Scheduling Block based system
  by end of 2005
• Up front planning required by observers at that
  point will significantly reduce burden on
  scientific staff
• Next year, can turn focus to usability issues,
  Science Program level, and managing the
  dynamically scheduled queue (as planned in 2004)