Remote Observing with the Keck Telescopes

1
Remote Observing with the Keck Telescopes

• Robert Kibrick, Director of Scientific Computing,
  University of California Observatories / Lick
  Observatory
• Science, Culture, and Education over Internet2
  Networks, April 4, 2001

2
Overview of Presentation

• Background
• The Keck Telescopes
• Mauna Kea Observatories
• Telescope Scheduling
• Modes of Observing
• Remote observing with the Keck Telescopes
• From Keck Headquarters in Waimea, Hawaii (32 km)
• From Santa Cruz, California via Internet2 (3200
  km)
• Operational models and issues
• Live videoconference with astronomers at Keck

3
The Keck Telescopes
4
The Keck Telescopes
5
Keck Telescope Facts

• Twin, 10-meter optical/infrared telescopes
• Largest telescopes of this type in the world
• Construction funded by W. M. Keck Foundation
• Observing time shared between 4 institutions
• California Institute of Technology (Caltech)
• University of California (UC)
• National Aeronautics and Space Adminstration
  (NASA)
• University of Hawaii (UH)
• Located atop 4,200 meter summit of Mauna Kea

6
Mauna Kea summit on the island of Hawaii
7
Mauna Kea Summit

• Dormant volcano
• Premier astronomy site in N. Hemisphere
• Above 90 of water vapor in atmosphere
• Non-turbulent airflow over the summit
• Sub-arcsecond atmospheric seeing
• Home to many international observatories

8
Mauna Kea Optical Observatories

• UH 0.6 meter
• UH 2.2 meter
• Gemini North
• CFHT
• NASA IRTF
• Keck-2
• Keck-1
• Subaru

9
Largest Optical Telescopes in the Northern
Hemisphere

• Hubble Space Telescope Mirror is similar in size
  to UH 2.2-meter
• Combined light gathering power of Mauna Kea
  telescopes is 50 times greater than HST

10
Common Facilities Operated by University of Hawaii

• Dormitories atHale Pohaku
• Altitude is 2,800m
• All water must be trucked in
• Acclimatization required before ascent to the
  4,200m summit

11
Mauna Kea Observatories Serve An International
Community

• Argentina
• Australia
• Brazil
• Canada
• Chile
• France
• Japan
• Netherlands
• Taiwan
• United Kingdom
• United States

12
Assigning Telescope Time Classical Scheduling

• Committee evaluates observing proposals
• Produce a 3 or 6 month telescope schedule
• Approved proposals assigned a set of dates
• Observing dates known many weeks in advance
• Airline tickets can be purchased at lower rates
• Astronomers can adjust course schedules
• Proposal writers conduct their own observations
• Can adjust observing program to weather
  conditions
• Can alter program in case of unexpected discovery

13
Assigning Telescope TimeQueue Scheduling

• Committee evaluates observing proposals
• Approved proposals placed in a queue
• Computer selects queue entries nightly
• Specific visibility or timing requirements
• Best match of current sky conditions
• Proposal priorities
• Observing dates not known in advance
• Proposal authors do not conduct observations

14
Queue scheduling works best if instruments can be
changed quickly

• Smaller, more recent telescopes, like the Subaru,
  have robotic mechanisms that permit instruments
  to be rapidly changed in the middle of the night.
• This flexibility makes queue scheduling easier to
  implement.

15
Keck instruments must be changed manually and
only during the day

• Instrument size scales with telescope size.
• Keck instruments are massive some weigh nearly
  8,200 kilograms.
• Keck instruments can only safely be changed
  manually and during the daytime.

16
Keck Telescopes use Classical Scheduling

• Kecks not designed for queue scheduling
• Schedules cover a semester (6 months)
• Approved proposals get 1 or more runs
• Each run is between 0.5 to 4 nights long
• Gaps between runs vary from days to months
• Half of all runs are either 0.5 or 1 night long
• Separate schedules for the two Kecks

17
Keck Telescope Schedule
18
Non-queue Scheduled Observing Modes

• Local Observer
• Controls observation from the telescope site
• Remote Observer
• Controls observation from a remote site
• Service Observer
• Observes in place of a remote observer
• Remote observer submits detailed object list
• Service observer conducts observation locally

19
Keck Telescope Observing Modes

• Modes neither supported nor planned
• Service observing
• Queue scheduled observing
• Supported observing modes
• Local observing
• Remote observing
• Astronomers who are granted observing time
  conduct their own observations

20
From 1993 to 1995, all Keck observing was done at
the summit

• Observers at the summit work from control rooms
  located adjacent to the telescope domes

21
Conducting observations involves coordinated
effort by 3 groups

• Telescope operator (observing assistant)
• Responsible for telescope safety operation
• Keck employee normally works at summit
• Instrument scientist
• Expert in operation of specific instruments
• Keck employee works at summit or Waimea
• Observers
• Select objects and conduct observations
• Employed by Caltech, UC, NASA, UH, or other

22
Keck 2 Control Room at the Mauna Kea Summit

• Telescope operator, instrument scientist, and
  observers work side by side, each at their own
  computer.

23
Observing at the Mauna Kea summit is both
difficult and risky

• Oxygen is only 60 of that at sea level
• Lack of oxygen reduces alertness
• Observing efficiency significantly impaired
• Altitude sickness afflicts some observers
• Some are not even permitted on summit
• Pregnant women
• Those with heart or lung problems

24
Initiative to support remote observing from Keck
Headquarters

• 1995 Remote control rooms built at Keck HQ
• Initial tests via 1.5 Mbps (T1) link to the
  summit
• 1996 Videoconferencing connects both sites
• Remote observing with Keck 1 begins
• 1997 gt50 of Keck 1 observing done remotely
• Link to the summit upgraded to 45 Mbps (DS3)
• 1999 remote observing gt90 for Keck 1 and 2
• 2000 remote observing is now the default mode

25
Keck 2 Remote Control Room at the Keck
Headquarters in Waimea

• Observer and instrument scientist in Waimea use
  video conferencing system to interact with
  telescope operator at the summit

26
Videoconferencing has proved vital for remote
observing from Waimea

• Visual cues (body language) important!
• Improved audio quality extremely valuable
• A picture is often worth a thousand words
• Chose dedicated versus PC-based units
• Original (1996) system was PictureTel 2000
• Upgrading to Polycom Viewstations

27
Keck 2 Remote Observing Room as seen from the
Keck 2 summit

• Telescope operators at the summit converse with
  astronomer at Keck HQ in Waimea via the
  videoconferencing system.

28
The Remote Observing Facility at Keck
Headquarters in Waimea

• Elevation of Waimea is 800 meters
• Adequate oxygen for alertness
• Waimea is 32 km NW of Mauna Kea
• 45 Mbps fiber optic link connects 2 sites
• A remote control room for each telescope
• Videoconferencing for each telescope
• On-site dormitories for daytime sleeping

29
The Keck Headquarters in Waimea

• Most Keck technical staff live and work in
  Waimea. Allows direct contact between observers
  and staff. Visiting Scientists Quarters (VSQ)
  located in same complex.

30
Limitations of Remote Observing from Keck HQ in
Waimea

• Most Keck observers live on the mainland.
• Mainland observers fly gt 3,200 km to get to
  Waimea
• Collective direct travel costs exceed 400,000
  U.S. / year

31
Remote Observing from Waimea is not cost
effective for short runs

• Round trip travel time is 2 days
• Travel costs gt 1,000 U.S. per observer
• About 50 of runs are for 1 night or less
• Cost / run is very high for such short runs
• Such costs limit student participation

32
Motivations for Remote Observing from the U.S.
Mainland

• Travel time and costs greatly reduced
• Travel restrictions accommodated
• Sinus infections and ruptured ear drums
• Late stages of pregnancy
• Increased options for
• Student participation in observing runs
• Large observing teams with small budgets
• Capability for remote engineering support

33
Fast and reliable network needed for mainland
remote observing

• 1997 1.5 Mbps Hawaii -gt Oahu -gt mainland
• 1998 10 Mbps from Oahu to mainland
• 1999 First phase of Internet-2 upgrades
• 45 Mbps commodity link Oahu -gt mainland
• 45 Mbps Internet-2 link Oahu -gt mainland
• 2000 Second phase upgrade
• 35 Mbps Internet-2 link from Hawaii -gt Oahu
• Now 35 Mbps peak from Mauna Kea to mainland

34
Internet-2 links
35
Mainland remote observing goals and
implementation strategy

• Goals
• Target mainland facility to short duration runs
• Avoid duplicating expensive Waimea resources
• Avoid overloading Waimea support staff
• Strategy
• No mainland dormitories observers sleep at home
• Access existing Waimea support staff remotely
• Restrict mainland facility to experienced
• Restrict to mature, fully-debugged instruments

36
Mainland remote observing facility is an
extension of Keck HQ facility

• Only modest hardware investment needed
• Workstations for mainland remote observers
• Network-based videoconferencing system
• Routers and firewalls
• Backup power (UPS) especially in California!!!
• Backup network path to Mauna Kea and Waimea
• Avoids expensive duplication of resources
• Share existing resources wherever possible
• Internet-2 link to the mainland
• Keck support staff and operational software

37
Keck software is accessed the same regardless of
observers location

• The control computers at the summit
• Each telescope and instrument has its own
  computer
• All operational software runs only on these
  computers
• All observing data written to directly-attached
  disks
• Users access data disks remotely via NFS or
  ssh/scp
• The display workstations
• Telescopes and instruments controlled via X GUIs
• All users access these X GUIs via remote displays
• X Client software runs on summit control
  computers
• Displays to X server on remote display
  workstation

38
Why did we choose this approach?

• Operational Simplicity
• Operational control software runs only at the
  summit
• All users run identical software on same computer
• Simplifies management between independent sites
• Allowed us to focus on commonality
• Different sites / teams developed instrument
  software
• Large variety of languages and protocols were
  used
• BUT all instruments used X-based GUIs

39
Focus effort on X standardization and
optimization over long links

• Maintain consistent X environment between sites
• Optimize X performance between sites
• Eliminates need to maintain
• Diverse instrument software at multiple sites
• Diverse telescope software at multiple sites
• Coordinate users accounts at multiple sites
• Fewer protocols for firewalls to manage

40
Accessing Keck software and data from Keck HQ in
Waimea

• Telescope operator uses display workstation at
  summit.
• Instrument scientist and observers use display
  workstations in Waimea.

41
Accessing Keck software and data from the mainland

• Telescope operator uses display workstation at
  summit.
• Instrument scientist uses display in Waimea
• Observers use display on mainland

42
Remote observing differences Waimea versus the
mainland

• System Management
• Keck summit and HQ share a common domain
• Mainland sites are autonomous
• Remote File Access
• Observers at Keck HQ access summit data via NFS
• Observers on mainland access data via ssh/scp
• Propagation Delays
• Summit to Waimea round trip time is about 1 ms.
• Summit to mainland round trip time is about 100
  ms.

43
Increased propagation delay to mainland presents
challenges

• Initial painting of windows is much slower
• But once created, window updates fast enough
• All Keck applications display to Waimea OK
• A few applications display too slowly to mainland
• System and application tuning very important
• TCP window-size parameter (Web100 Initiative)
• X server memory and backing store
• Minimize operations requiring round trip
  transactions

44
Tradeoffs from this approach to remote observing

• Disadvantages
• X protocol does not make optimal use of bandwidth
• Long propagation delays require considerable
  tuning
• Advantages
• Minimizes staffing requirements at mainland sites
• Only vanilla hardware and software needed there
• Simplifies sparing and swapping of equipment
• Simplifies system maintenance at mainland sites

45
End-to-end reliability is critical to successful
remote operation

• Keck Telescope time is valued at 1 per second
• Each observer gets only a few nights each year
• Observers wont use facility if not reliable
• What happens if network link to mainland fails?
• Path from Mauna Kea to mainland is long complex
• At least 14 hops crossing 7 different network
  domains
• While outages are rare, consequences are severe
• Even brief outages cause session collapse panic
• Observing time loss can extend beyond outage

46
Mitigation plan install end-to-end ISDN-based
fallback path

• Install ISDN lines and routers at
• Each mainland remote observing site
• Keck 1 and Keck 2 control rooms
• Fail-over and fallback are rapid and automatic
• Toll charges incurred only during network outage
• Lower ISDN bandwidth reduces efficiency, but
• Observer retains control of observations
• Sessions remain connected and restarts avoided
• Prevents observer panic

47
Summary of ISDN-based fallback path

• Install 3 ISDN (6 B channels) between sites
• Install Cisco 2600-series routers at each end
• Dual auto-sensing Ethernet interfaces
• Quad BRI interfaces
• Inverse multiplexing
• Dial-on-demand (bandwidth-on-demand)
• Caller ID (reject connections from unrecognized
  callers)
• Multilink PPP with CHAP authentication
• Uses GRE tunnels and OSPF routing
• No manual intervention needed at either end

48
Summary of progress to date

• Prototype mainland facility assembled at UCSC
• Multipoint videoconferencing to summit and HQ
• Efficiently displays telescope instrument
  status
• Two Keck instruments operated remotely
• High Resolution Spectrometer (HIRES) on Keck 1
• Echellette Spectrogram Imager (ESI) on Keck 2
• Primary use has been for remote engineering
• On-sky remote observing performed with ESI
• Efficient, automated transfer of image files

49
Future Plans

• Conduct ongoing trials using prototype at UCSC
• Work out operational details with Keck staff
• Add capability to operate additional instruments
• Aim for fully operational status by late 2001
• Extend to other sites once debugged at UCSC

50
Items Currently in Progress

• Real time display of guide camera images
• Installation of ISDN lines at the Keck summit
• Installation of ISDN routers at the Keck summit
• Adjustments to telescope scheduling procedures
• Firewall issues at summit and mainland sites

51
Summary

• Internet-2 makes mainland operation feasible
• Proposed model should be affordable
• Mainland sites operate as satellites of Waimea
• Leverage investment in existing facilities and
  staff
• Leverage investment in existing software
• Share existing resources wherever feasible
• Maintain Waimea as focal point for remote
  operation
• Avoid wasteful duplication of resources
• Avoid expensive and inefficient travel for short
  runs

52
Mainland remote observing gives observers a choice

• Yields significant advantages for short runs
• Provides alternatives when travel is difficult
• Increases options for student participation
• Increases options for large observing teams
• Provides option of multi-site collaborations
• Waimea remains available for longer runs

53
Acknowledgments

• U.S. National Science Foundation
• U.S. Department of Defense
• University of Hawaii
• Gemini Telescope Consortium
• University Corp. for Advanced Internet
  Development (UCAID)
• Corporation for Education Network Initiatives in
  California (CENIC)

54
Author Information

• Robert Kibrick, UCO/Lick Observatory
• University of California, Santa Cruz
• California 95064, U.S.A.
• E-mail kibrick_at_ucolick.org
• WWW http//www.ucolick.org/kibrick
• Phone [phone]
• FAX [phone]

55
www.internet2.edu