Utilising Large

1
Utilising Large Astronomical Datasets
Brian Schmidt The Research School of Astronomy
and Astrophysics Mount Stromlo Siding Spring
Observatories
2
The Golden Age of Astronomy

• The last decade has seen Astronomers
• Measure the age of the Universe
• Discover the first extra-solar planet
• Map the Cosmic Neighbourhood
• Figure out how much and what makes up the
  Universe
• Get a good idea on the ultimate fate, and
  beginning of the Universe

3
What is Left to do?

• Observing and understanding the first generation
  of Stars and Galaxies
• Directly detecting and studying Extrasolar
  Planets
• What is the Dark Energy?
• What is the Dark Matter?

4
(No Transcript)
5
1 Billion Dollars
6

Astronomy Cannot Afford to Live on these
Instruments Alone

• Technology provides the ability to create large
  datasets
• Sift through rare events, or objects, or add the
  information
• of a billion objects together, rather than
  observing a few objects very
• well.
• To Succeed, you must have
• Well chosen Scientific Goals
• Carefully chosen Experimental Plan
• Need to work with and beyond Cutting Edge
  technology both hardware and software.

7
Worked Example- A large Optical Survey of The
Southern Sky
Observing and understanding the first generation
of Stars and Galaxies Find the Brightest
First Galaxies (Quasars) in the sky, and study
these. Challenge only about 10 useful objects in
the sky for the current 8m class telescopes.
8
Worked Example- A large Optical Survey of The
Southern Sky
Directly detecting and studying Extrasolar
Planets- ChallengeNeed to monitor millions of
stars on the timescale of hours to see the
eclipse
Venus 2004
Venus 1882
9
Worked Example- A large Optical Survey of The
Southern Sky
What is the Dark Energy? As photons from the
Cosmic Microwave Background travel through the
gravity wells of galaxies, they compress on the
way in, and stretch, on the way out this
process is asymmetric in one of the two favourite
forms of Dark Energy. Challenge, need more than a
billion galaxies with which to compare to get
enough statistics
10
Worked Example- A large Optical Survey of The
Southern Sky
What is Dark Matter? Find the most distant set
of stars in our own galaxy, and use them as
tracers of the gravity. How far does the Dark
Matter Extend, is it spherical? Challenge, need
to sift out the roughly 5000 stars that are
bright enough to be studied in the outer reaches
of our Galaxy.
11

• Great Melbourne Telescope
• Equipped with state-of-the art CCDs to
• Provide 32 Million Pixels of data every 90
  seconds
• Over 5 years, a 5 colour 24 Terabyte Image of the
  entire southern sky at 3 epochs

12
(No Transcript)
13
Great Melbourne Telescope automated 2000

• Installing Weather Monitoring system (15000 2
  people months)
• Installing computer controllable switches on all
  systems (15000 2 people months)
• Moving from Liquid Nitrogen to Closed Cycle
  Cryogenic System for Instrument cooling (30000
  4 people months)
• Control Software to monitor weather and control
  telescope (1 person year)
• Software Scheduler to decide what to observe (2
  person months)
• Quality Monitoring software (2 person months)
• New Imaging system (350000 2 people years)

14
(No Transcript)
15
A new Southern Survey
Designed from ScratchSo Huge Field of View7
square degrees (50 times full moon) 300 Million
Pixels of information every 70 seconds Entire
sky in 10 nights Over 2 years, A map of the
entire sky in 6 colours, with information at
approximately 10 epochs. Available to the
entire community.
16
The New Telescope
17
Survey Size Over 2 years 140,000 300,000,000
pixel images 115 Terabytes of information
18
Factor of 10 compression, noise increased by 5
19
140,000 300,000,000 pixel images 115 Terabytes
of information but can store this onto Terabytes (40,000 for a disk array of this
size) Or store on a dedicated Mass storage system
(e.g. ANU/APAC Supercomputing facilitys Terabyte
storage array)
20

• Robust Pipelines
• Where in the sky is this?
• Where are the objects?
• How bright are the objects?
• What shape are the objects?
• What is bad information?
• Beg, Borrow, and Steal from The past 20 years of
  Astronomical work. No black boxes, nothing
  proprietary!

21
Data Analysis Flow Current Pipeline takes
approximately 1 hour to complete on a 2.5GHz
processor on a single 300,000,000 pixel
image. Fortunately we have nighttime and bad
weather! On A long winters night 500 images
would be taken, So total computational power
required for analysis is 22 2.5GHz processors
(roughly another 40,000) In 3 years time only
8 of the newly released 8GHZ Intel Tritium (they
are hot!) processors will be required.
22

• Potential Problems
• Funding small potatoes compared to big
  telescopes, but still big bickkies for grants
  available to Australian Researchers
• Automated systems and instruments are non-trivial
  to implement. It takes a lot of time of effort to
  make them trouble free.
• Software although computers can handle the 113
  Terabytes, humans cannot. Must have robust
  pipelines that work completely without human
  intervention.

23
There are scientific opportunities to those who
are willing to tackle large data sets. But
requires carefully planned scientific outcomes
like any other experiment. Usually Requires
Researchers to design/modify the data collecting
hardware as well as the software which they use
for experiments. No one person (or even group)
has all of this knowledge But Universities are
full of expertise where there is an idea, there
is most certainly a way.