Astronomy, Petabytes, and MySQL

1

• Astronomy, Petabytes, and MySQL
• MySQL Conference
• Santa Clara, CA
• April 16, 2008
• Kian-Tat Lim
• Stanford Linear Accelerator Center

2
Outline

• LSST
• LSST Database
• LSST Database MySQL

3
LSST

• What Is It?
• Why Build It?

4
LSST

• What Is It?
• Why Build It?

5
Telescope

• Proposed telescope to be built in Chile

6
Large
3.2 gigapixel camera
8.4 meter diameter mirror
7
Synoptic Survey

• Wide
• Deep
• Fast

8
LSST

• What Is It?
• Why Build It?

9
Dark Matter and Energy
Photo J. A. Tyson, W. Colley, E. L. Turner, and
NASA
10
Variable Objects
11
Transient Objects
12
Moving Objects
Photo D. Roddy, Lunar and Planetary Institute
13
LSST Database

• Whats In It?
• How Big?
• How Often?
• What Queries?
• Unusual Needs

14
LSST Database

• Whats In It?
• How Big?
• How Often?
• What Queries?
• Unusual Needs

15
Database Components
Moving ObjectsCatalog
Object Catalog
Provenance Statistics Summaries
Source Catalog
Difference Image Source Catalog
Image Metadata
Calibration
Engineering and Facility Database
16
Astronomical Objects
Moving ObjectsCatalog
Object Catalog
Provenance Statistics Summaries
Source Catalog
Difference Image Source Catalog
Image Metadata
Calibration
Engineering and Facility Database
17
Sources
Moving ObjectsCatalog
Object Catalog
Provenance Statistics Summaries
Source Catalog
Difference Image Source Catalog
Image Metadata
Calibration
Engineering and Facility Database
18
Changes
Moving ObjectsCatalog
Object Catalog
Provenance Statistics Summaries
Source Catalog
Difference Image Source Catalog
Image Metadata
Calibration
Engineering and Facility Database
19
Image Metadata
Moving ObjectsCatalog
Object Catalog
Provenance Statistics Summaries
Source Catalog
Difference Image Source Catalog
Image Metadata
Calibration
Engineering and Facility Database
20
Calibration and Facility
Moving ObjectsCatalog
Object Catalog
Provenance Statistics Summaries
Source Catalog
Difference Image Source Catalog
Image Metadata
Calibration
Engineering and Facility Database
21
LSST Database

• Whats In It?
• How Big?
• How Often?
• What Queries?
• Unusual Needs

22
Sagans of Rows

• 49 billion objects
• 2.8 trillion sources

23
Lots of Columns

• 308 columns for objects
• 56 columns for sources
• (for now)

24
Database Size

• Grows to gt14 PB

25
LSST Database

• Whats In It?
• How Big?
• How Often?
• What Queries?
• Unusual Needs

26
Frequency

• Nightly updates
• Semi-annual data releases

27
LSST Database

• Whats In It?
• How Big?
• How Often?
• What Queries?
• Unusual Needs

28
Queries

• All about an object
• All objects meeting criteria
• All objects near objects meeting criteria
• All objects with interesting time series
• All pairs of objects with similar time series

29
LSST Database

• Whats In It?
• How Big?
• How Often?
• What Queries?
• Unusual Needs

30
Unusual Needs

• Flexibility
• Provenance

31
LSST Database MySQL

• Why MySQL?
• Scalability?
• Performance?

32
LSST Database MySQL

• Why MySQL?
• Scalability?
• Performance?

33
MySQL

• Relational database management system

34
Open Source

• Vibrant community
• Strong company support

35
Hardware

• Runs on commodity hardware

36
In-Memory Tables

• Needed for near-real-time processing

37
LSST Database MySQL

• Why MySQL?
• Scalability?
• Performance?

38
MySQL Grid
39
Partitioning

• Large tables partitioned spatially

40
Replication

• Dimension tables likely replicated

41
Needs Distributor/Combiner

• LSST will build prototype
• Need long-term support

42
LSST Database MySQL

• Why MySQL?
• Scalability?
• Performance?

43
Per-Column Indexing

• 2X data size

44
Needs Optimizer

• Efficient use of multiple (20-30) indexes

45
Needs Indexes

• Bitmap/compressed indexes

46
Needs Storage Engine

• Shared scan for long-running full-table queries

47
Summary

• Building a petabyte DB
• MySQL can be a core component