Implementing Unified Access to Scientific Data from .NET Platform

1
Implementing Unified Access to Scientific
Datafrom .NET Platform
Moscow State University Department of
Computational Mathematics and Cybernetics

• Sergey B. Berezin
• Dmitriy V. Voitsekhovskiy
• Vilen M. Paskonov

Supported by Student Laboratory of Microsoft
Technologies and RFBR grants
2
Different languages, common tools
Viscous fluid flow visualization via vector
fields and color maps (http//www.cs.msu.su)
Seismic data visualization via isosurfaces
(http//www.sci.utah.edu)
Tensor field visualization for diffusion through
biological tissue (http//www.sci.utah.edu)
3
Scientific data access requirements

• We need to
• Retrieve typed data object
• without regard where it is stored
• and how it is stored.
• Physical data independence
• Retrieve partial data when needed
• Filtering Caching
• Retrieve data description
• Metadata support

Seismic data visualization via isosurfaces
(http//www.sci.utah.edu)

• We dont want to
• Rewrite existing
• computational software
• Use existing formats
• Install new system software
• Use existing protocols

4
Scientific data access today
5
Whats so special in scientific data?

• Scientific data

• Have a complex structure
• Parameterized by
• Time
• Sampling point coordinates
• More complex parameters
• Stored in many files of various formats
• Have very large size of individual data items
• Dont fit well to relational model!

6
What is DataSet?
7
Example Accessing data in C

• // Retrieve the DataSet object from a server by
  GUID
• DataSet dataset DataSet.Open("http//regatta.cs.
  msu.su9111", "767c57b1-801e-4784-bbd6-287707fd0ec
  2")
• // Fetching DataItem by name.
• // DataItem may be either simple or composite, it
  doesnt matter
• DataItem xVelocity dataset.DataItems"u-values"
  
• // Creating parameter corresponding to time
  moment 0.0
• CompositeParameter param new CompositeParameter(
  
• new ParameterValue("time", 0.0d) )
• // Fetching DataItemSlice for the parameter.
• // It is an instance of DataItem for specified
  parameter value.
• DataItemSlice dataVelocity xVelocityparam
• // Getting required data velocity array for time
  0.0
• ScalarArray3d data dataVelocity.GetData() as
  ScalarArray3d

8
DataRequest communicating with server

• The following DataRequest is sent to a server as
  the result of the previous example

ltsoapEnvelope gtltsoapBodygt ltdataRequest
dataSource""
dataSet"767c57b1-801e-4784-bbd6-287707fd0ec2"
gt ltdataItem typeScalarArray3d"gt
ltdataSource sourceName"u0000.cdf"
sourceType"netCDF"
sourceParameters"u" /gt
lt/dataItemgt lt/dataRequestgt lt/soapBodygtlt/soapEnv
elopegt

• The following DataRequest is received from the
  server

ltsoapEnvelope gtltsoapBodygt ltdataRequest
dataSource"" dataSet"767c57b1-801e-478
4-bbd6-287707fd0ec2" gt ltdataItem
typeScalarArray3d"gt ltdataRef
sourceTypeu0000.cdf" sourceParametersu"gt
ltremote url"scp//regatta.cs.msu.su/datasource/pv
m/Re1000/u0000.cdf /gt lt/dataRefgt
lt/dataItemgt lt/dataRequestgt lt/soapBodygtlt/soapEnve
lopegt
9
Complex structures in DataSet
x
file with scalar array
scalar array data item
X,Y,Z
file with scalar array
scalar array data item
vector array data item
file with scalar array
scalar array data item
vector array constructor
vector field data item
file with spatial grid
spatial grid data item
data field constructor
10
Example Accessing composite data in C

• // Retrieve the DataSet object from a server by
  GUID
• DataSet dataset DataSet.Open("http//regatta.cs.
  msu.su9111", "767c57b1-801e-4784-bbd6-287707fd0ec
  2")
• // Fetching DataItem by name.
• // DataItem may be either simple or composite, it
  doesnt matter
• DataItem velocity dataset.DataItems"uvw-values"
  
• // Creating parameter corresponding to time
  moment 0.0
• CompositeParameter param new CompositeParameter(
  
• new ParameterValue("time", 0.0d) )
• // Fetching DataItemSlice for the parameter.
• // It is an instance of DataItem for specified
  parameter value.
• DataItemSlice dataVelocity velocityparam
• // Getting required data velocity array for time
  0.0
• Vector3dArray3d data dataVelocity.GetData() as
  Vector3dArray3d

11
DataRequest composite data items

• The following DataRequest is sent to a server as
  the result of the previous example execution

ltsoapEnvelope gtltsoapBodygt ltdataRequest
dataSource"" dataSet"767c57b1-801e-4784-bbd6-28
7707fd0ec2" gt ltdataItem type"Vector3dArray3d"
gt lt!- This is a COMPOSITE DATAITEM! --gt
ltcomposite constructor"CompositeVectorArray"gt
ltcomponentgt lt!-- u-values --gt ltdataItem
type"ScalarArray3d"gt ltdataSource
sourceName"u0000.cdf" sourceType"netCDF"
sourceParameters"u" /gt lt/dataItemgt
lt/componentgt ltcomponentgt lt/componentgt lt!--
v --gt ltcomponentgt lt/componentgt lt!-- w --gt
lt/compositegtlt/dataItemgtlt/dataRequestgt lt/soapBodygt
lt/soapEnvelopegt

• The following DataRequest is received from server

ltsoapEnvelope gtltsoapBodygt ltdataRequest
dataSource"" dataSet"767c57b1-801e-4784-bbd6-28
7707fd0ec2" gt ltdataItem typeVector3dArray3d"
gt ltdataRef sourceType"plain binary"
sourceParameters""gt ltremote
url"scp//regatta.cs.msu.su/datasource/Fdg4gBd
/gt lt/dataRefgt lt/dataItemgt lt/dataRequestgt lt/so
apBodygtlt/soapEnvelopegt
12
Filtering

• Filtering allows transfer of only required data
  from server to client
• Filtering may be performed both by a client-side
  and a server-side of the system.
• Examples of the filtering are cropping and
  thinning of large vector fields.

cropping filter 0.4,0.76 x 0.4,0.76
thinning filter (0.1,0.1)
0.76
1
100 x 100 2d vectors 120KB
3000 x 3000 2d vectors 108 ?B
0.4
0.76
1000 x 1000 2d vectors 12 MB
0.4
0.76
0
1
0.4
0.4
0.76
13
Example Filtering data in C

• // Initializing the DataSet object from a server
  by its GUID
• DataSet dataset DataSet.Open("http//regatta.cs.
  msu.su9111", "767c57b1-801e-4784-bbd6-287707fd0ec
  2")
• // Fetching DataItem by its name. It may be
  either simple or composite
• DataItem velocity dataset.DataItems"uvw-values"
  
• // Creating parameter corresponding to time
  moment 0.0
• CompositeParameter param new CompositeParameter(
  
• new ParameterValue("time", 0.0d) )
• // Fetching DataItemSlice for the parameter.
• DataItemSlice dataVelocity velocityparam
• // Creating filter "Thinner" for a type of the
  velocity data item
• // and setting up its parameters
• IThinner3dFilter filter FilterFactory.GetFilter(
  "Thinner", dataVelocity.TypeDescriptor) as
  IThinner3dFilter
• filter.PercentageX 0.05
• filter.PercentageY 0.05
• filter.PercentageZ 0.05
• // Getting required data thinned out velocity
  array for time 0.0
• Vector3dArray3d data dataVelocity.GetData(filter
  ) as Vector3dArray3d

14
DataRequest communicating with server

• The following DataRequest is sent to a server as
  the result of the previous example execution

ltsoapEnvelope gtltsoapBodygt ltdataRequest
dataSource"" dataSet"767c57b1-801e-4784-bbd6-28
7707fd0ec2" gt ltfilter name"Thinner"gt
ltparametersgt ltparameter namePercentageX
value0.05 typedouble /gt ltparameter
namePercentageY value0.05 typedouble /gt
ltparameter namePercentageZ value0.05
typedouble /gt lt/parametersgt ltdataItem
type"Vector3dArray3d"gt lt!- This is a COMPOSITE
DATAITEM! --gt ltcomposite constructor"Composi
teVectorArray"gt ltcomponentgt lt!-- u-values
--gt ltdataItem type"ScalarArray3d"gt
ltdataSource sourceName"u0000.cdf"
sourceType"netCDF" sourceParameters"u" /gt
lt/dataItemgt lt/componentgt ltcomponentgt
lt/componentgt lt!-- v --gt ltcomponentgt
lt/componentgt lt!-- w --gt lt/compositegtlt/dataItemgt
lt/filtergt lt/dataRequestgt lt/soapBodygtlt/soapE
nvelopegt

• The returned DataRequest is similar in this case
  to the returned DataRequest from the previous
  example.

15
Caching

• Both server-side and client-side of the system
  cache the results of a successful DataRequest
  execution.
• Server-side cache filtering results
• Client-side cache retrieved data items and
• results of DataRequest

16
How DataRequest is performed?
17
Deployment Scenario

• The simplest scenario is as follows

• More sophisticated scenario includes development
  of distributed data sources that provide
  scientific data.
• Dedicated servers will act as data processor
  performing data filtering and transformations
• Dedicated servers will act as data registries
  allowing DataSet enumeration and querying in
  entire global network.
• This will make possible to create dynamic data
  libraries of researches and enables easy data
  publishing

18
Why .NET?

• Object-oriented data access requires an
  object-oriented platform to be built on.
• High extensibility is based on CLR dynamic nature
• New data types
• New filters
• New parsers
• No built in data types, filters, parsers
• .NET opens new horizons with LINQ, WPF,

19
Future work

• Transferring files from remote server is just an
  example of DataProvider
• Extend architecture for new types of data
  providers
• LINQ technology will make data access from C
  much more elegant.
• Development of easy-to-use data management
  applications for the proposed approach.
• Development of an innovative visualization
  system, highly extensible and customizable
• Or integrate our approach with existing one

20
Future visualization system
Step 1. Choose object of interest Step 2. Choose
data transform Step 3. Choose visualization
algorithm
Example
21
Questions?

• Visit http//microsoft.cs.msu.su/projects/uvs
• Mail to s_berezin_at_cs.msu.su