Hello

1
Hello!
2
Virtual Observatories for the Developing
WorldAjit Kembhavi, IUCAA
3
The Data Avalanche
Immense amounts of data are being produced by
large telescopes using large area detectors.
Terabytes of data are now available, and
Petabytes will soon be available from frequent
all sky imaging.
Vast databases are also being produced through
simulations.
4
Wavelength Coverage, Resolution
The data spans the electromagnetic spectrum from
the radio to the gamma-ray region.
Obtaining, analysing and interpreting the data in
different wavebands and at different resolutions
involves highly specialised instruments and
techniques.
The astronomer needs new holistic tools for using
this wealth of data.
5
(No Transcript)
6
Virtual Observatories

• Develop interoperability concepts to make
  different databases seamless.
• Manage vast data resources and provide these
  on-line to astronomers and other users.
• Provide tools for data discovery, access,
  analysis, visualization and mining.
• Provide data storage and grid computing
  facilities.
• Empower astronomers, regardless of their
  location and circumstances, to make an impact on
  research and development.

New Science Initiatives
7
IVOA Technology Initiatives

• The IVOA has identified six major
  technical initiatives to fulfill the scientific
  goal of the VO concept

Develop databases and tools consistent with this
framework
Registries, Data models, Uniform Content
Descriptors, Data Access Layers,
VO Query Language, VOTable

8
(No Transcript)
9
(No Transcript)
10

• Provide Russian astronomers effective access to
  international resources
• Integrate Russian astronomical resources into the
  international VO structure
• Provide access to observational resources where
  the needed data is not in archives
• Develop electronic educational resources

ADS, Vizier, INES, Hyperleda
Science Projects Three dimensional map of
interstellar extinction in the Galaxy (with
NVO) Fundamental stellar parameters and
evolutionary status of close binaries (with
Besenscon) Open clusters, MIGALE, Stellar
evolutionary parameters through astrophysical
tracks
GLASS LIBRARIES
11
Virtual Observatory - India
A collaboration between IUCAA and PSPL, with a
grant from the Ministry of Communications and
Information Technology
12
VO-India Software Projects
VOPlot Visualizer for catalogue data VOTable
C Parser VOTable Streaming writer
Data Converters Fits
Browser User interfaces and
query tools Applications
beyond astronomy All tools have web-based and
stand alone versions
VOStat
13
The VOPlot Tool

• A VO-I CDS collaboration
• First conceived as a web-based tool for Vizier
• Then integrated with Aladin
• VOPlot is now also a stand alone system
• It has been integrated with many data
  bases
• VOPlot now has several faces

VOPLot VOPLot3D VOMegaPLot
14
(No Transcript)
15
Colour-magnitude diagram
parallax
16
On-the-fly GUI

Back
VOPlot, Aladin, SIMBAD, NED, VOStat
Jayant Gupchup, Mohasin Sheikh
17
Mohasin Shaikh, Deoyani, PSPL team
18
(No Transcript)
19
The Digital Divide in Astronomy-

• Immense databases, electronic archives of
  scientific periodicals are available free.
• The latest research is available through
  preprints.
• Virtual Observatory tools will make all this
  highly accessible and usable.

20
But-

• Many astronomers lack the bandwidth, expertise
  and the environment to make use of these riches
• There is resistance to the use of new concepts
  and tools
• There are reservations about exposing data

21
(No Transcript)
22
Bridging the Gap

• The Indian experience could be easily replicated
  elsewhere in the developing world
• Astronomers can lead the charge
• Astronomers in the developing world could help
  build archives, develop software and provide much
  needed human resources
• The Third World Astronomy Network can provide a
  platform

23
Virtual Observatory - India
VOPlot VOPlot3D VOMegaPLot VOStat
VOConvert VOCat
24
Thank You
25

• REGISTRIES collect metadata about data
  resources and information services into a
  queryable database. The registry is distributed.
  
• UNIFORM CONTENT DESCRIPTORS These will
  provide the common language for for metadata
  definitions for the VO.

VOTable This is an XML mark-up standard for
astronomical tables.
26
Himalaya Chandra Telescope Data Archives
27
(No Transcript)
28
SDSS J125637-022452 High proper motion L-subdwarf
Optical spectra of mixed late M and mid L
type Only the third L subdwarf known
29
VOPlot-Aladin interoperability
Object highlighted
Point highlighted
30
Positions 1986-2000
Proper motion 0.617 arcsec / yr
31
Discovery of Optically Faint Obscured Quasars
Padovani et al AA 2004

• VO techniques were used to identify obscured
  quasars from the GOODS fields.
• X-ray sources were selected on the basis of their
  luminosity and hardness ratio.
• These were cross correlated with optical sources,
  which were studied using GOODS image cutouts from
  Aladin.
• 31 new Type 2 quasars were discovered compared to
  the 9 previously known.

32
Virtual Observatory
-
India
-TNG
New Projects VOStat, VOEvent Rolling sky maps
(think Google maps) Grid computing for
morphology Visualization and Analysis
Theoretical VO
A collaboration between IUCAA and PSPL, with a
grant from the Ministry of Communications and
Information Technology
33
Results in Aladin

Back
34
Data Storage and Retrieval
electromagnetic spectrum from the
The Astronomer Vermeer 1632-1675
The Library of Alexandria 3rd
Century BC
35
Data Explosion
Peter Quinn
36
VOPlot - Introduction

• Tool for visualizing astronomical data.
• Developed in JAVA
• Plots data available in the VOTable format.
• Available as stand alone version and web based
  version (integrated with VizieR).
• Uses Ptplot 5.2, a 2D data plotter and histogram
  tool implemented in Java.

37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
Results in VOPlot

Back
41
Query using a Form

Back
42
Query using SQL Directly

Back
43
(No Transcript)
44
(No Transcript)
45
Some Data Interface Tools screenshots
46
(No Transcript)
47
(No Transcript)
48
Catalog Data Interface Tool

• A tool to query catalog data.
• Simple, customizable, graphic interface.
• Not specific to type of data or
  catalogue.
• SQL queries for expert users.
• VO tools available for
  analysis
• VOPlot, Aladin, SIMBAD, NED, VOStat
• Jayant Gupchup, Mohasin Sheikh

49
Data Organization and Architecture
50

• REGISTRIES These collect metadata about
  data resources and information services into a
  queryable database. The registry is distributed.
  A variety of industry standards are being
  investigated.
• DATA MODELS This initiative aims to define
  the common elements of astronomical data
  structures and to provide a framework to describe
  their relationships.
• UNIFORM CONTENT DESCRIPTORS These will
  provide the common language for for metadata
  definitions for the VO.

51

• DATA ACCESS LAYER This provides a
  standardized access mechanisms to distributed
  data objects. Initial prototypes are a Cone
  Search Protocol and a simple Image Access
  Protocol.
• VO QUERY LANGUAGE This will provide a
  standard query language which will go beyond the
  limitations of SQL.
• VOTable This is an XML mark-up standard for
  astronomical tables.

52
Astronomical Data Explosion
100 Gb/night
P. Quinn
53
Persistent Systems Pvt. Ltd., Pune
54
Virtual Observatory - India
55
Data Archives and Mirrors at VO-I
SDSS 2Mass 2DFGRS
2QZ
FIRST
NVSS
Chandra
Vizier, Aladin, ADS
56
Fast ComputingIUCAA Resource
Eight alpha server ES-45 nodes, each
with 4 processors, each node with 8 GB
RAM Fast, low latency interconnect
Memory Channel Architecture Trucluster
clustering environment (Tru64 Unix, DecMPI,
openMP)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
Stars in the Milky Way
62
The Hertzsprung-Russell Diagram
63
The Hertzsprung-Russell Diagram
64
Virtual Observatory -India
A collaboration between IUCAA and PSPL, with a
grant from the Ministry of Communications and
Information Technology
65
Science Initiatives

• Many IVOA projects have active Science Working
  Groups consisting of astronomers from a broad
  cross-section of the community representing all
  wavelengths.
• The focus here is to develop a clear perception
  of the scientific requirements of a VO.
• Projects within the working groups will develop
  new capabilities for VO based analysis.
• This will enable the community to create new
  research programs and to publish their data and
  research in a more pervasive and scientifically
  useful manner.

66
AVO Prototype Demo Astrogrid Astronomy Catalogue
Extractor AVO AladinSED VO-IndiaVOPlot
67
FITS Manager
View, create and add to FITS files Convert to
other formats Pallavi Kulkarni
Fits-manager
68
VOTable Java Streaming Writer
Acts on a data array in memory to convert it to
the VOTable form, which is streamed row by
row to an output file. Very large VOTables can
be written without excessive memory. Pallavi
Kulkarni
VOTable-Java
69
VOTable

• This is a new data exchange standard produced
  through efforts led by Francois Ochsenbien of
  CDS, Strasbourg and Roy Williams of Caltech.
• VOTable is in XML format. Physical quantities
  come with sophisticated semantic information.

70
VOTable

• The format enables computers to easily parse the
  information and communicate it to other
  computers.
• Federation and joining of information become
  possible and Grid computing is easier.
• VOTable parsers have been developed in Perl, Java
  and C.
• Enhancements and extensions are being considered.

Streaming Parser
Non-streaming Parser
71
VOTable Data

• The data part in a VOTable may be represented
  using one of three different formats
• FITS VOTable can be used either to encapsulate
  FITS files, or to re-encode the metadata.
• BINARY Supported for efficiency and ease of
  programming, no FITS library is required, and the
  streaming paradigm is supported.
• TABLEDATA Pure XML format for small tables.

72
C VOTable Parser

• Motivation
• Provide a library for API based access to VOTable
  files.
• APIs can be directly used to develop VOTable
  applications without having to do raw VOTable
  processing.
• Streaming and Non-streaming versions are
  available.

Sonali Kale, Sudip Khanna
73
C VOTable Parser

• Salient Features
• Implemented as a wrapper over XALAN-C.
• XALAN-C is a robust implementation of the W3C
  recommendations for
• XSL Transformations (XSLT) and the
• XML Path language (XPath).
• XPath queries can be used to access the VOTable
  data.

74
Project Design
VTable
Metadata
Link Collection
Link
Field
Field Collection
Link
Link Collection
Values
Table Data
minimum
Row Collection
maximum
Row
Option Collection
Column Collection
Options
75
IUCAA HPC Facility Hercules

• HPC Team
• Sarah Ponthratnam
• Sunu Engineer
• Rajesh Nayak
• Anand Sengupta

• Co-proposed by
• Ajit Kembhavi
• T. Padmnabhan
• Tarun Souradeep

• Four Alpha Server ES-45 machines
• Each with 4 processors Alpha (21264C)
• 1.25 GHz clock speed
• Cache on chip 64 Kb I, 64 Kb-D
• Cache 16 Mb ECC DDR
• RAM 3 x 8 Gb 12 Gb
• Fast, Low latency interconnect
• Memory channel Architecture (MCA)
• High volume Storage
• 1 Tera-byte SCSCI
• Trucluster clustering environment (Tru64 Unix,
  DecMPI, openMP)

gt 30 G flops Preliminary HPL benchmark
ES-45 Specfp2000 1327 Linpack 1000x1000 6847
76
Virtual Observatory - India
Persistent Systems
IUCAA
77
Virtual Observatory-India
Ajit Kembhavi IUCAA
Anand Deshpande Persistent Systems
Funded by the Ministry of ICT and Persistent
78
Caltech, Fermilab, JHU, NASA/HEARC, Microsoft,
NCSA/UIUC, NOAO, NRAO, Raytheon ITS, SDSC/UCSD,
SAO/CXC, STScI, UPenn, UPitts/CMU, UWis, USC,
USNO, USRA, CVO

• NVO-People

79
Virtual Observatory - India
Ajit Kembhavi Inter-University Centre for
Astronomy and Astrophysics Pune, India

80
Virtual Observatories

• Provide tools for data analysis, visualization
  and mining.
• Develop interoperability concepts to make
  different databases seamless.
• Manage vast data resources and provide these
  on-line to astronomers and other users.
• Empower astronomers by providing sophisticated
  query and computational tools, and computing
  grids for producing new science.

81
Terapix
Jodrell Bank
82
Registry and DIS
83
High Volume Storage
Raid 5, 4 Terabyte
84
CVO Collaborations

• There are three major projects at the CVO
  involving collaborations with other VO.
• CVO is collaborating with the German
  Astrophysical VO to incorporate ROSAT X-ray data
  and catalogues into the CVO system.
• CVO is collaborating with the Australian VO.to
  incorporate 2Qz and 2DF galaxy spectra into the
  CVO database.
• CVO is an associate member of NVO and is have put
  in place some components of the NVO galaxy
  morphology demo.

85
Science Initiatives

• Many IVOA projects have active Science Working
  Groups consisting of astronomers from a broad
  cross-section of the community representing all
  wavelengths.
• The focus here is to develop a clear perception
  of the scientific requirements of a VO.
• Projects within the working groups will develop
  new capabilities for VO based analysis.
• This will enable the community to create new
  research programs and to publish their data and
  research in a more pervasive and scientifically
  useful manner.

86
Australian VO Collaborations

• The distributed volume renderer (dvr) software,
  is a tool for rendering large volumetric data
  sets using the combined memory and processing
  resources of Beowulf like clusters.
• A collaboration between the Melbourne site of
  Aus-VO and AstroGrid aims to develop the existing
  dvr software into a grid-based volume rendering
  service.
• Users will be able to select FITS-format cubes
  from a number of "Data Centres",have the data
  transferred to a chosen rendering cluster, and
  then proceed to visualise the volume of data
  remotely (See Demo).

87
C VOTable Parser

• Initial version
• - Released on May 31st , 2002.
• - Support only for reading of tables.
• - Support only for pure-XML TABLEDATA and not
  for BINARY or FITS data streams.
• - Runs on Windows NT 4.0, Windows 2000 and
• RedHat Linux 7.1.
• Future enhancements
• - Can be incorporated quickly and
  efficiently.

88
Parser Design

• Class Details
• VTable In memory representation of a single
  ltTABLEgt
• from the ltRESOURCEgt element in VOTable
• TableMetaData Contains MetaData (Fields, Links
  and Description)
• Resource Represents the ltRESOURCEgt element in
  the VOTable.
• TableData Contains Rows
• Field Representation of ltFIELDgt from VOTable
• Row Representation of ltTRgt from VOTable
• Column Representation of ltTDgt from VOTable

89
Parser Design

• API Typical Operations
• File Level I/O Routines
• Open VOTable file
• Close VOTable file
• Table I/O Operations
• Get number of rows
• Get number of columns
• Get column(field) information (column name,
  column number, etc.)
• Accessing table data

90
Parser Implementation

• Development on Windows NT 4.0 platform using
  VC. Ported to RedHat Linux 7.1/gcc-2.96 with
  zero effort.
• 18 C classes representing various elements of
  the VOTable format.
• 8500 lines of C code written for V1.1 release
• Project start date April 7th 2002
• V1.1 Release May 31st 2002
• Current status V1.2 design in progress

91
What is in Release V1.1

• Parser to serve as a building block for
  developing VOTable based applications.
• Can be easily used by users of CFITSIO library.
• Supports powerful XPath queries against VOTable
  files.
• The first version of the VO Table parser can now
  be downloaded
• http//vo.iucaa.ernet.in/voi/html/infopage.h
  tml

92
VOTable Parser Demo

• Serves as a tutorial to help understand the basic
  APIs provided by the VOTable parser.
• Demonstrates how to access the data and metadata
  elements of a VOTable file.

93
Future Work

• Develop APIs for writing data in VOTable format.
• Develop APIs for supporting IMAGE data and FITS
  files in VOTable.
• Enhance existing API set to allow more elaborate
  and flexible operations on VOTable files.
• Support future VOTable versions.
• Develop applications for conversion between FITS
  and VOTable formats.

94
References

• The first version of the C parser can now be
  downloaded from the VO-India website
• http//vo.iucaa.ernet.in/voi
• VOTable Details
• http//vizier.u-strasbg.fr/doc/VOTable/
• XALAN
• http//xml.apache.org/xalan-c/index.html
• XPATH
• http//www.w3.org/TR/xpath

95
(No Transcript)
96
(No Transcript)
97
Virtual Observatory - India
98
(No Transcript)
99
(No Transcript)
100
Star Positions
101

• REGISTRIES These collect metadata about data
  resources and information services into a
  queryable database. The registry is distributed.
  A variety of industry standards are being
  investigated.
• DATA MODELS This initiative aims to define the
  common elements of astronomical data structures
  and to provide a framework to describe their
  relationships.
• UNIFORM CONTENT DESCRIPTORS These will provide
  the common language for for metadata definitions
  for the VO.

102
(No Transcript)
103
(No Transcript)
104
VO Schema
105
Browse Server Database

Back
106
Open Cluster Membership
Pleiades POSS II image
107
Loading catalogues
parallax
108
Plotting Data Parallax Histogram
cluster parallax 8.460.22 mas
109
Colour-magnitude diagram
ZAMS
can correct for reddening by adding a new column
110
Colour-magnitude diagram
parallax