Astronomical data curation and the WideField Astronomy Unit

1
Astronomical data curation and the Wide-Field
Astronomy Unit

• Bob Mann
• Wide-Field Astronomy UnitInstitute for
  AstronomySchool of PhysicsUniversity of
  Edinburgh(rgm_at_roe.ac.uk)

2
Outline

• Who we are
• Introduction to the Wide-Field Astronomy Unit
• What we do
• Sky survey data curation past, present and
  future
• Data curation and the Virtual Observatory
• What we could do with you
• What WFAU could do for the DCC
• What the DCC could do for WFAU
• Questions

3
Outline

• Who we are
• Introduction to the Wide-Field Astronomy Unit
• What we do
• Sky survey data curation past, present and
  future
• Data curation and the Virtual Observatory
• What we could do with you
• What WFAU could do for the DCC
• What the DCC could do for WFAU
• Questions

4
Wide-Field Astronomy Unit

• Funded to curate optical and near-infrared sky
  survey data for UK (and European) community
• Based at Royal Observatory Edinburgh
• 35 years of sky survey data curation at ROE
• Evolving data holdings
• Photographic plates
• Digital scans of photographic plates
• Born-digital data
• WFAU formed in 1999 group moved into UoE
• Currently 12 grant-funded 2 academic staff
• Mix of astronomers, IT professionals hybrids

5
Outline

• Who we are
• Introduction to the Wide-Field Astronomy Unit
• What we do
• Sky survey data curation past, present and
  future
• Data curation and the Virtual Observatory
• What we could do with you
• What WFAU could do for the DCC
• What the DCC could do for WFAU
• Questions

6
Sky survey data life-cycle e.g. WFCAM

• Images taken at telescope
• UKIRT, in Hawaii
• Data reduction pipeline run in Cambridge
• Removes instrumental signatures
• Produces final, clean images
• Detects and characterises sources in images
• Data transferred to Edinburgh
• Ingest source catalogues and image metadata into
  relational database, store image files on disk
• Combine data from multiple nights new images,
  cats.
• Publish release databases via web interface

On pernight basis
7
WFAUs main survey archives

• Past SuperCOSMOS
• Based on digital scans of photographic plates
• Database 5TB largest tables 109 rows
• Images 35,000 user requests (10GB) per month
• Present (2005-2012) WFCAM
• Near-infrared 700 registered users
• 500 million rows of database results per month
• 125GB of flat file image data per month
• Near-future (2008-2020) VISTA
• 3 x data rates/volume of WFCAM

8
WFAUs future plans

• Large Synoptic Survey Telescope
• US-led public/private project
• Were trying to get UK to buy into it
• Data challenges immense
• WFCAM takes 20TB of image data per year
• LSST will take 20TB of image data per
  night60PB images, 8PB database (2016-2025)
• LSST stimulating a lot of data management RD in
  the US
• Commercial Google
• Academic Sci-DB (M. Stonebraker, D. DeWitt)

9
The Virtual Observatory

• Goal an interoperable federation of all the
  worlds astronomical data resources
• International Virtual Observatory Alliance
• Coordinates VO development worldwide
• Acts as W3C-like standards body for the VO
• AstroGrid
• Only project to have developed a full VO system

10
Virtual Observatory components

• Registry
• Metadata for all data published to the VO
• Standard data access protocols
• For tabular data, images, spectra, time series,
  etc
• Standard web service wrappers for application
  code
• Enabling asynchronous calls, workflow, etc
• Distributed data storage system
• Presenting transparent aggregated logical view to
  user

11
Curation challenges for WFAU

• More data analysis services in the data centre
• Data volumes too large for user download
• WFAU must provide data analysis services
  hardware
• Integration of data and knowledge
• Third-party annotations which can be used in
  queries
• Object X in database Y is a quasar
• X-ray source A is the same object as radio
  source B
• Better linkage between archives and online
  literature
• Keeping staff up to date on technologies/technique
  s
• Mostly learn by doing do we make best choices?

12
Outline

• Who we are
• Introduction to the Wide-Field Astronomy Unit
• What we do
• Sky survey data curation past, present and
  future
• Data curation and the Virtual Observatory
• What we could do with you
• What WFAU could do for the DCC
• What the DCC could do for WFAU
• Questions

13
WFAU and DCCWhat we can do for you

• Case studies, exemplars, etc
• WFAU is a well-established, competent group
• Astronomy is a relatively small, cohesive
  community, used to interdisciplinary
  collaboration
• Astronomers are early adopters of IT and
  recognise value of data curation
• VO is a rich, functional e-Science infrastructure
• Collaborations to date
• Raj Bose distributed annotation service
• James Cheney paper on data centre security

14
WFAU and DCCWhat you can do for us

• Policy advice
• Increasingly need to convince research councils
  of benefits of long term data curation
  cost/benefit
• Technical advice from DCC or its Associates
• Should we use iRODS for LSST?
• Do any XML databases have decent performance?
• Do the VO metadata standards make sense?
• Curation manual
• When will the rest appear?
• Training
• e.g. NeSC course on relational database design

15
WFAU and DCCQuestions

• What is the DCCs model for collaboration?
• Cant collaborate with everyone on everything
• Scientists digital librarians live in different
  worlds how do you bridge that divide?
• Interdisciplinary work requires sustained
  interaction
• What do you want from scientific data curators?
• What can you offer us in return?
• Few of my colleagues know anything about the DCC
• Does that surprise you?