SkyServer: Public Access to the Sloan Digital Sky Survey

1
SkyServer Public Access to the Sloan Digital Sky
Survey
Sigmod 2002, Madison

• Alex Szalay, Jim Gray, Ani Thakar, Peter Kunszt,
  Tanu Malik, Tamas Budavari, Jordan Raddick, Chris
  Stoughton, Jan vandenBerg

2
Outline

• The Sloan Digital Sky Survey
• The SDSS database design
• HTM spatial queries
• 20 queries
• Demo of the SkyServer
• The next steps
• The World-Wide Telescope
• Web Services
• Sky Query/Image Cutout

3
Features of the SDSS
Goal Create the most detailed map of
the Northern sky in 5 years 2.5m telescope,
Apache Point, NM 3 degree field of view ¼
of the whole sky Two surveys in one
Photometric survey in 5 bands Spectroscopic
redshift survey Automated data reduction 150
man-years of development Very high data volume
40 TB of raw data 5 TB processed catalogs
Data is public
The University of Chicago Princeton
University The Johns Hopkins University The
University of Washington New Mexico State
University Fermi National Accelerator
Laboratory US Naval Observatory The
Japanese Participation Group The Institute for
Advanced Study Max Planck Inst, Heidelberg
Sloan Foundation, NSF, DOE, NASA
4
The Imaging Survey
Continuous data rate of 8 Mbytes/sec Northern
Galactic Cap drift scan of 10,000 square
degrees 24k x 1M pixel panoramic
images in 5 colors broad-band
filters (u,g,r,i,z) exposure time 55 sec
pixel size 0.4 arcsec astrometry 60 mas
calibration 2 done only in best seeing
(20 nights/year) Southern Galactic Cap
multiple scans (gt 30 times) of the same
stripe
5
The Spectroscopic Survey
Elliptical galaxy
Expanding universe redshift
distance SDSS Redshift Survey 1 million
galaxies 100,000 quasars 100,000 stars Two high
throughput spectrographs spectral range 3900-9200
Å 640 spectra simultaneously R2000 resolution,
1.3 Å Features Automated reduction of
spectra Very high sampling density and
completeness
6
Data Flow
7
SDSS Data Products
Object catalog 6000 GB parameters of gt108
objects Redshift Catalog 1 GB
parameters of 106 objects Atlas Images 1500 GB
5 color cutouts of gt108 objects Spectra
60 GB in a one-dimensional form Derived
Catalogs 20 GB clusters QSO absorption
lines 4x4 Pixel All-Sky Map 60 GB heavily
compressed Corrected Frames 15 TB
8
Spatial Data Access SQL extension

• Szalay, Kunszt, Brunner http//www.sdss.jhu.edu/ht
  m
• Added Hierarchical Triangular Mesh (HTM)
  table-valued function for spatial joins
• Every object has a 20-deep Mesh ID
• Given a spatial definition,routine returns up to
  10 covering triangles
• Spatial query is then up to 10 range queries
• Very fast 10,000 triangles / second / cpu

9
20 Queries

• DB design started with 20 queries in English
• These then dictated DB design
• Spatial extensions, neighbors
• Then implemented in SQL
• Heavy use of SP, UDF
• All run in 10 mins, most under 1 min
• Tag tables
• replaced by covering indices
• Sequential IO
• The worst case, a full scan reached 400MB/sec
  on Wintel

10
Q15 Fast Moving Objects

• Find near earth asteroids

SELECT r.objID as rId, g.objId as gId,
dbo.fGetUrlEq(g.ra, g.dec) as url FROM PhotoObj
r, PhotoObj g WHERE r.run g.run and
r.camcolg.camcol and abs(g.field-r.field)lt2
-- nearby -- the red selection criteria and
((power(r.q_r,2) power(r.u_r,2)) gt 0.111111
) and r.fiberMag_r between 6 and 22 and
r.fiberMag_r lt r.fiberMag_g and r.fiberMag_r lt
r.fiberMag_i and r.parentID0 and r.fiberMag_r lt
r.fiberMag_u and r.fiberMag_r lt
r.fiberMag_z and r.isoA_r/r.isoB_r gt 1.5 and
r.isoA_rgt2.0 -- the green selection
criteria and ((power(g.q_g,2) power(g.u_g,2))
gt 0.111111 ) and g.fiberMag_g between 6 and 22
and g.fiberMag_g lt g.fiberMag_r and
g.fiberMag_g lt g.fiberMag_i and g.fiberMag_g lt
g.fiberMag_u and g.fiberMag_g lt g.fiberMag_z and
g.parentID0 and g.isoA_g/g.isoB_g gt 1.5 and
g.isoA_g gt 2.0 -- the matchup of the pair and
sqrt(power(r.cx -g.cx,2) power(r.cy-g.cy,2)power
(r.cz-g.cz,2))(10800/PI())lt 4.0 and
abs(r.fiberMag_r-g.fiberMag_g)lt 2.0
11
Demo of SkyServer

• Based on the TerraServer design
• Designed for high school students
• Contains 150 hours of interactive courses
• Experiment for easy visual interfaces
• Opened June 5, 2001
• After a year
• 1.6M page views
• 60K visitors
• 4.7M page hits
• Added Web Services
• Cutout
• SkyQuery

http//skyserver.sdss.org/
12
Public Data Release

• June 2002 EDR
• Early Data Release
• January 2003 DR1
• Contains 30 of final data
• 100 million photo objects
• 4 versions of the data
• Target, best, runs, spectro
• Total catalog volume 1.7TB
• See Terascale sneakernet paper
• Published releases served forever
• EDR, DR1, DR2, .
• O(N2) only possible because of Moores Law!

EDR
13
Why Is Astronomy Data Special?

• It has no commercial value
• No privacy concerns
• Can freely share results with others
• Great for experimenting with algorithms
• It is real and well documented
• High-dimensional (with confidence intervals)
• Spatial
• Temporal
• Diverse and distributed
• Many different instruments from many different
  places and many different times
• The questions are interesting
• There is a lot of it (petabytes)

14
Living in an Exponential World

• Astronomers have a few hundred TB now
• 1 pixel (byte) / sq arc second 4TB
• Multi-spectral, temporal, ? 1PB
• They mine it looking for new (kinds of) objects
  or more of interesting ones (quasars),
  density variations in 400-D space correlations
  in 400-D space
• Data doubles every year
• Data is public after 1 year
• So, 50 of the data is public
• Some have private access to 5 more data
• So 50 vs 55 access for everyone

15
Virtual Observatory

• Many new surveys are coming
• SDSS is a dry run for the next ones
• LSST will be 1TB/night
• All the data will be on the Internet
• But how? ftp, webservice
• Data and apps will be associated withthe
  instruments
• Distributed world wide
• Cross-indexed
• Federation is a must, but how?
• Will be the best telescope in the world
• World Wide Telescope

16
SkyQuery Experimental Federation

• Federated 5 Web Services
• Portal unifies 3 archives and a cutout service to
  visualize results
• Fermilab/SDSS, JHU/FIRST, Caltech/2MASS Archives
• Multi-survey spatial join and SQL select
• Distributed query optimization (T. Malik, T.
  Budavari) in 6 weeks
• http//www.skyquery.net/
• Cutout web service annotated SDSS images
• http//SkyService.jhu.pha.edu/SdssCutout

SELECT o.objId, o.ra, o.r, o.type, t.objId FROM
SDSSPhotoPrimary o, TWOMASSPhotoPrimary t
WHERE XMATCH(o,t)lt3.5 AND AREA(181.3,-0.76,6.5)
AND o.type3 AND o.I t.m_j gt 2
17
Relevant Papers

• Data Mining the SDSS SkyServer DatabaseJim Gray
  Peter Kunszt Donald Slutz Alex Szalay Ani
  Thakar Jan Vandenberg Chris Stoughton Jan. 2002
  40 p.
• An earlier paper described the Sloan Digital Sky
  Surveys (SDSS) data management needs Szalay1
  by defining twenty database queries and twelve
  data visualization tasks that a good data
  management system should support. We built a
  database and interfaces to support both the query
  load and also a website for ad-hoc access. This
  paper reports on the database design, describes
  the data loading pipeline, and reports on the
  query implementation and performance. The queries
  typically translated to a single SQL statement.
  Most queries run in less than 20 seconds,
  allowing scientists to interactively explore the
  database. This paper is an in-depth tour of those
  queries. Readers should first have studied the
  companion overview paper The SDSS SkyServer
  Public Access to the Sloan Digital Sky Server
  Data Szalay2.
• SDSS SkyServerPublic Access to Sloan Digital Sky
  Server DataJim Gray Alexander Szalay Ani
  Thakar Peter Z. Zunszt Tanu Malik Jordan
  Raddick Christopher Stoughton Jan Vandenberg
  November 2001 11 p. Word 1.46 Mbytes PDF 456
  Kbytes
• The SkyServer provides Internet access to the
  public Sloan Digital Sky Survey (SDSS) data for
  both astronomers and for science education. This
  paper describes the SkyServer goals and
  architecture. It also describes our experience
  operating the SkyServer on the Internet. The SDSS
  data is public and well-documented so it makes a
  good test platform for research on database
  algorithms and performance.
• The World-Wide TelescopeJim Gray Alexander
  Szalay August 2001 6 p. Word 684 Kbytes PDF 84
  Kbytes
• All astronomy data and literature will soon be
  online and accessible via the Internet. The
  community is building the Virtual Observatory, an
  organization of this worldwide data into a
  coherent whole that can be accessed by anyone, in
  any form, from anywhere. The resulting system
  will dramatically improve our ability to do
  multi-spectral and temporal studies that
  integrate data from multiple instruments. The
  virtual observatory data also provides a
  wonderful base for teaching astronomy, scientific
  discovery, and computational science.
• Designing and Mining Multi-Terabyte Astronomy
  Archives Robert J. Brunner Jim Gray Peter
  Kunszt Donald Slutz Alexander S. Szalay Ani
  ThakarJune 1999 8 p. Word (448 Kybtes) PDF (391
  Kbytes)
• The next-generation astronomy digital archives
  will cover most of the sky at fine resolution in
  many wavelengths, from X-rays, through
  ultraviolet, optical, and infrared. The archives
  will be stored at diverse geographical locations.
  One of the first of these projects, the Sloan
  Digital Sky Survey (SDSS) is creating a
  5-wavelength catalog over 10,000 square degrees
  of the sky (see http//www.sdss.org/). The 200
  million objects in the multi-terabyte database
  will have mostly numerical attributes in a 100
  dimensional space. Points in this space have
  highly correlated distributions.
• The archive will enable astronomers to explore
  the data interactively. Data access will be aided
  by multidimensional spatial and attribute
  indices. The data will be partitioned in many
  ways. Small tag objects consisting of the most
  popular attributes will accelerate frequent
  searches. Splitting the data among multiple
  servers will allow parallel, scalable I/O and
  parallel data analysis. Hashing techniques will
  allow efficient clustering, and pair-wise
  comparison algorithms that should parallelize
  nicely. Randomly sampled subsets will allow
  de-bugging otherwise large queries at the
  desktop. Central servers will operate a data pump
  to support sweep searches touching most of the
  data. The anticipated queries will re-quire
  special operators related to angular distances
  and complex similarity tests of object
  properties, like shapes, colors, velocity
  vectors, or temporal behaviors. These issues pose
  interesting data management challenges.

18
References and Links

• SkyServer
• http//skyserver.sdss.org/
• http//research.microsoft.com/pubs/
• Virtual Observatory
• http//www.us-vo.org/
• http//www.voforum.org/
• World-Wide Telescope
• paper in ScienceV.293 pp. 2037-2038. 14 Sept
  2001. (MS-TR-2001-77 word or pdf.)
• SDSS DB is a data mining challenge
• Get your personal copy athttp//research.microsof
  t.com/gray/sdss