ORR Readiness

1
ORR Readiness

• Mark Colavita Peter Wizinowich
• KISWG
• Pasadena 10/29/2003

2
Question to Address (2/3/2003 pre-ORR
presentation)

• At the ORR
• Will the instrument be capable of achieving the
  requirements in the PLRA at the start of routine
  science operations? (V2 mode)
• What are the relevant performance metrics and
  current performance? (V2 mode)
• Today
• What are the specific performance requirements,
  and how do they relate to the PLRA?
• How will compliance be demonstrated?
• Are they technically achievable?
• What is the current performance?
• What is the plan to get from current performance
  to ORR? Is the plan realistic?
• What are the risks for routine science operation?
  How will they be mitigated? Are there additional
  risks not yet identified that need to be
  addressed?

3
Specific Performance Requirements (2/3/2003
pre-ORR presentation)

• PLRA requirements for V2
• System Performance Goals
• Provide 5 rms accuracy V2 amplitude measurements
  for targets as faint as K9 using the two Keck
  telescopes
• Provide at least 90 uptime for the
  interferometer and 90 uptime for the two 10m
  telescopes and their AO systems
• System Performance Requirement
• Provide at least 90 uptime for the
  interferometer and 80 uptime for the two 10m
  telescopes and their AO systems

Program Level Requirements Appendix to Navigator
Program Plan
4
Pre-ORR Testing from Oct Observing Run

• Targets included measurements of
  well-characterized PTI binary
• Used neutral density filters to attenuate this
  bright star to simulate K9 ORR requirements
• Data accuracy still being assessed
• Observation
• K9 spec will need to be qualified with a V or R
  magnitude, as stars faint to AO are less well
  corrected, and provide less flux into fringe
  tracker
• (Fringe tracker measures flux Strehl)
• Plan to provide this for ORR
• Also tests of fringe tracker low-V2 tracking limit

5
Oct Data(Most processing still in-progress)
6
Prior (May, Jan) accuracy tests
7
Other tests
8
Provide at least 90 uptime for the
interferometer and 90 uptime for the two 10m
telescopes and their AO systems (2/3/2003 pre-ORR
presentation)

• Quoted directly from the PLRA
• NB not a measure of open shutter time or
  efficiency, which is actually quite low for any
  instrument, like IF, that is calibration-limited
• A measure equal to 100 minus the downtime
  required to fix IF hardware or software problems
  during scheduled science observing
• Telescopes/AO addressed in a few slides
• Demonstrating compliance
• Mostly proper bookkeeping
• Identify a timekeeper to track system state vs.
  time
• Demonstrate compliance on two runs prior to ORR
  with good weather and median or better seeing

9
System uptime scans per hour
10
Uptime summary

• Interferometer uptime ? 90 2/5 runs
• Facility uptime ? 90 1/5 runs ? 80 3/5 runs
• October run problem areas
• Major
• Telescope control system crashes
• K2 continued to have problems after the IF run
  supporting DEIMOS
• Problem may be resolved need to test
• Minor most in process of being addressed
• Facility
• AO offload problem (possible interaction from IF
  angle tracker)
• K2 AO Wavefront controller crashes
• Interferometer
• Sequencer crashes
• Aligner crate reboots
• Telemetry server memory growth solved
• Goal is to achieve ORR metric performance on next
  two runs

11
Performance, functionality, efficiency goals
(2/3/2003 pre-ORR presentation)

• These follow from the desire to maximize the
  science achievable at the specified performance
  level

12
Performance, functionality, efficiency goals, 1
(2/3/2003 pre-ORR presentation)
13
System uptime scans per hour
14
Scans per hour

• 86 science plus engineering scans in 22.5 clear
  hours
• Peak scans on each of 4 nights
• 2, 3, 6, 8
• Sustained high throughput impacted by
• System faults
• Ease of observing target
• Faint targets, requiring optimization of AO, KAT,
  FATCAT, or which end up not being trackable,
  count against scans per hour
• Weather (light to moderate cirrus, which doesnt
  count against weather loss)
• May need to qualify our 6 scan/hr goal to apply
  to sources not pushing system limits

Night 3 of Oct run
Peak scans/hr for this night
15
Performance, functionality, efficiency goals,
1updates to 2/3/2003 pre-ORR presentation
16
Performance, functionality, efficiency goals, 2
17
Performance, functionality, efficiency goals, 3
18
Procedural Requirements for the ORR

• From the PLRA
• Provide an operational infrastructure including
  trained staff, facilities, hardware and software,
  so that the Keck Interferometer will operate as a
  facility for use by the community scientists.

19
Procedural Requirements for the ORR

• From LaPiana Need (A) requirements which you
  can track against your progress to actually
  achieve the metric performances, and the (B)
  ability to do it routinely and having an
  organization in place that is well trained to do
  so with robust and stable software (at WMKO
  MSC).
• Questions to address at the ORR from LaPiana
• Will the ground data system (at WMKO and at MSC)
  implementation and testing be complete prior to
  the start of routine science operations?
• Will the required operations plans and
  procedures, including those for anomaly
  responses, be completed and tested prior to start
  of routine science operations?
• Are the operations organization, staff and
  facilities in place (WMKO and MSC)?
• Will the necessary training activities be
  completed prior to start of routine science
  operations?
• Are the necessary set of commands and sequences
  ready to support nominal operations? Will all
  action items, liens, etc., be satisfactorily
  dispositioned?
• Are the risks well understood and acceptable for
  routine science operations?

20
1. Ground Data System Completion

• Will the ground data system (at WMKO and at MSC)
  implementation and testing be complete prior to
  the start of routine science operations?
• At the pre-ORR The primary demonstration will be
  successful CARA operation of the interferometer
  for V2 science.
• We will track this performance over the next 7
  runs.
• The last 3-runs prior to the ORR will be
  performed at the full ORR requirement level.
• CARA personnel have successfully operated the
  interferometer for most of these runs. We have
  tracked performance during these runs. Many, but
  not all, of performance numbers demonstrated in
  Oct. run.
• At most can demonstrate full ORR-level compliance
  during only 2 runs (Nov. Jan.) prior to ORR.

21
2. Operations Plans Procedures

• Will the required operations plans and
  procedures, including those for anomaly
  responses, be completed and tested prior to start
  of routine science operations?
• Operations procedures for daytime checkout
  night-time operation are in place.
• Anomaly responses need completion
• Including spares troubleshooting training
• Configuration management (software,
  opto-mechanical electronics) need to be defined
  agreed upon
• Especially important during ongoing development
• Major software upgrade will happen with RTC 2.1
• Also need upgrade plan (plan to implement
  upgrades without compromising operational
  system).
• Would be easier if some subsystems had already
  been handed over to CARA
• Need to approve/implement plan for maintaining
  cleanliness of optics/basement

22
2. Operations Plans Procedures

• Current operations procedures
• Punchlist with new priorities distributed after
  each run. Reviewed bi-weekly.
• Schedule for summit activities maintained.
  Reviewed weekly. Incorporated in Observatory
  scheduling.
• Daytime summit activities entered into IFDEV
  Observatory daylog.
• Observing support personnel (daytime, nighttime
  on-call) list distributed prior to run.
• Setup observing procedures available on-line
  (next slide).
• Afternoon pre-observing checklist/procedure
  performed.
• Plan for nights observing distributed.
• Observing log maintained published to telescope
  nightlog.
• Post-run summary distributed, including
  priorities for next run.
• Punchlist with old priorities closed out.
• Post-run review held with Observatory/Project
  management.

23
OperationsProcedures
24
34. Operations Organization Training

• Are the operations organization, staff and
  facilities in place (WMKO and MSC)?
• Will the necessary training activities be
  completed prior to start of routine science
  operations?
• Organization
• John Gathright has assumed Operations Coordinator
  role
• Responsible for coordinating/scheduling/staffing
  activities in support of observing runs,
  observing prep daytime experiments
• 2 (of 3) technicians report directly to him

25
34. Operations Organization Training

• Facilities
• Control rooms set up at headquarters summit
• Control rooms routinely used for interferometer
  operation
• Instrument performance capabilities need to be
  made available to the community for observing
  planning

26
34. Operations Organization Training

• Staff
• 3 Interferometer specialists trained in operation
  of Interferometer for daytime alignments,
  checkouts tests, night-time operations
• 2 technicians trained in daytime alignments
  checkouts
• Plan to hire an interferometer operator an
  interferometer specialist to relieve operational
  load on above personnel
• Software engineers (2) able to support day
  night-time troubleshooting.
• Electronics engineer able to support daytime
  troubleshooting.
• Mechanical engineer gaining some instrument
  familiarity.

27
34. Operations Organization Training

• Staff
• Currently 1 JPL 1 MSC person participates in
  each run. Can we fill their roles after the ORR?
  
• JPL person really helps with troubleshooting
• We are gaining more familiarity with instrument
  resolving problems (JPL still on-call)
• MSC person really understands science priorities
• We need more insight /or MSC on-call

28
5. Commands Sequences

• Are the necessary set of commands and sequences
  ready to support nominal operations? Will all
  action items, liens, etc., be satisfactorily
  dispositioned?
• Alignment
• Procedures exist and are used.
• All mirrors and shutters needed are remotely
  controlled.
• Pre-Observing checklist
• Checklist exists and routinely used by CARA
  personnel.
• Observing procedure/checklist
• CARA personnel routinely operate.
• MSC tools procedures
• CARA personnel recently operating.
• Post-Observing
• Tapes written and shipped.

29
5. Commands Sequences

• Target acquisition (Manual Automated steps)
• Interferometer Sequencer (IS) operator selects
  target from list
• IS loads parameters initiates Telescope
  Sequencer (TS)
• IS configures FDLs for observation
• TS initiates the following (can change for repeat
  observation)
• Telescope control system slews, then tracks, the
  telescopes
• Auto-acquisition tool positions brightest target
  on acquisition camera for AO system
• AO auto-setting tool optimizes AO parameters on
  target
• AO operator reports AO loops locked to IS
  operator
• AO system offloads tip/tilt and focus to the
  telescope
• IS operator closes Angle Tracker (KAT) loop
• KAT loop automatically offloads IF tip/tilt
  mirror to AO wavefront sensor offsets
• AO operator unloads offsets to field steering
  mirrors if gt threshold
• IS operator initiates IS to acquire fringes
• IS scans to peak signal on fringe tracker
• IS searches for fringes

30
5. Commands Sequences

• Remaining issues
• Automated night-time LDL repositioning (manual
  too slow)
• This is currently a high priority
• Automated daytime alignment (labor telescope
  intensive)
• Likely a lien at the ORR
• User interface is complex distributed (can make
  mistakes)
• Plan to implement a simpler quick-look
• Better version will come after RTC 2.1 upgrade
• Observing sequence automation should be improved
  to achieve scans/hour with reduced human
  interaction
• Selectable level of telescope sequencer operation
• Offloading wavefront sensor centroids to FSMs
• Reuse of AO parameters
• Remote configuration
• A few remaining hands-on operations

31
6. Risks

• Are the risks well understood and acceptable for
  routine science operations?
• Risks identified at ORR requirements review
• Weather, wind shake and seeing
• There are nights when IF observing simply cannot
  be done productively or when only bright targets
  can be used.
• Best choice in some conditions would be to switch
  to another instrument program that can succeed
  with poor seeing.
• Hardware failures
• Mitigation List of critical spares and backup
  approaches to be identified implemented prior
  to ORR.

32
6. Risks

• Development during operations
• Mitigation Configuration control.
• Observatory-wide software configuration plan
  currently being approved. Will apply to
  Interferometer AO ops.
• Inteng tag of intdev nearly in place to protect
  software.
• ECR process in place for AO opto-mechanics.
• Mitigation Careful coordination of development
  operations activities (using current operations
  procedures).
• Number of shared risk TAC allocated V2 science
  nights
• Too many nights could result in inadequate CARA
  involvement in development /or inadequate access
  to interferometer for development.
• Possible mitigation
• Limit the number of science nights.
• Additional operations staff.
• New Load on CARA staff if system requires lots
  of hand-holding

33
ORR readiness summary

• Will probably be OK on V2 accuracy spec need to
  complete analysis of Oct data
• Uptime still problematic
• Working on major (and minor) problems from Oct
• With satisfactory resolution of major, and
  several of the minor problems, should be possible
  to achieve in November Jan.
• Scans per hour still low, although brief good
  peak performance
• Some issues with brightness of target will not
  apply if target requires extensive optimization
• Improvements to system uptime and user interface
  should improve performance
• OK on most other areas, except for LDL
  automation, which is a current focus