Online data access in the SeaDataNet V1 system

1

• Online data access in the SeaDataNet V1 system
• by
• Dick M.A. Schaap technical coordinator
• Bologna, September 08

2
Common Data Index CDI

• CDI is a fine-grained index to individual data
  measurements (such as a CTD cast or moored
  instrument record). This is in contrast to EDMED,
  which is an inventory of data sets, each of which
  comprises a large number of measurements.
• CDI paves the way for direct online data access
  through file downloads or online data requests.
• Present CDI V0 system gt 340.000 CDI entries
  from 36 Data Centres from 29 countries
• http//seadatanet.maris2.nl/v_cdi_v0/search.asp

3
Common Data Index CDI V0
4
Common Data Index CDI V0
5
Common Data Index CDI V0
6
CDI V0

• Central and homogeneous metadatabase, but for
  data access the users are confronted with the
  various user interfaces of the SeaDataNet data
  centres (which can provide online search and
  access / ordering facility / e-mail request)

User interface
CDI
url
url
url
Data centre User interface
Data centre User interface
Data centre User interface
data
data
data
7
CDI V1

• Central and homogeneous metadatabase plus
  transparent access to the distributed data sets
  via a unique user interface at the SeaDataNet
  portal

User interface
CDI
Central User Register
Shopping basket
Request Status Manager
Download Manager
Download Manager
Download Manager
data
data
data
8
CDI V1 System Modules

• CDI Metadatabase and User Interface
• Searching and browsing of metadata of data sets,
  managed by SeaDataNet Data Centres
• SeaDataNet Central User Register with User Roles
  db web service
• Details of users, their organizations and
  addresses, license period, Id-Passwords,
  SeaDataNet Roles of users
• SeaDataNet Shopping Basket
• Preparing a user request of multiple data sets,
  handling validation of users and roles via the
  Central User Register, and routing requests to
  the Request Status Manager
• SeaDataNet Request Status Manager
• Processing and administration of all requests
  and data deliveries (downloads), for users in
  communication with data centres. For data
  centres to oversee all transactions.

9
System Modules

• SeaDataNet Data centre Download Manager
• Handles communication with SeaDataNet Request
  Status Manager and takes care that requested
  files are made ready for downloading by User (if
  cleared! ) in the agreed formats via a local
  website address.

search shop
Data download
SeaDataNet Portal
Data centre
Data centre
Data centre
Data centre
Metadata transaction data
10
Order flow diagram
Search in CDI directory
CDI Meta database
Register Users Licenses

Results
Requests list
Shopping Basket
User Register
User Roles
Submit Requests (specify filetype? )
Log in
WS
User Check
Info about Conditions, Registration etc
NO
YES User Role Requests
11

CDI V1 Search home page
12

Search Results page
13

Search Detail page (full)
14

Search records selected for Shopping basket
15

Shopping basket
16

Submit Shopping list gt User Authentication
17

User identified and request for motivation and
order confirmation
18

Request confirmed and link to RSM (also by e-mail)
19
YES User Role Requests

Shopping Basket
Submit extended Request Form with motivation
Requests user info in RSM Split by DC
Status Pending
Request Status Manager (RSM)
Confirmation email to User with RSM info
A B C D
U
V
W
Combi logic
Validation of access via business matrix

20
Data access restrictions in CDI V1
CC Collection cost charge
FE Commercial charge
LI Licence
MO Moratorium
NA No access
NC Distribution cost
OG Organisation
RS Restricted
SR Academic
UK Unknown
UN Unrestricted
See Vocabulary L081 for definitions Multiple
restrictions may apply
21
User Roles in AAA Register, of relevance for data
access
SDRN02 Public
SDRN03 Academic
SDRN04 Commercial
SDRN05 National and local government
SDRN06 Pan-National government

See Vocabulary C866 for definitions
Multiple roles may apply
22
Business Matrix
Restriction Role A B C D
U
V
W
TBD
TBD
NO
OK
OK
TBD
TBD
NO
NO
NO
OK
TBD
23
Request Status Manager (RSM)
Validation of access via business matrix
Data Centre
Download Manager

Status OK
Accepted requests in Download Manager
Status TBD
Status NO
E-mail to DC manager
Status OK
Retrieving files preparing formats (if
database) placing/updating daily ZIP file in
user Download directory notifying RSM of
ready for download status
Status Ready for Download
Validate in RSM
Access Denied
Status Access Denied
24
Request Status Manager (RSM)
Data Centre
Download Manager
Retrieving files preparing formats (if
database) placing/updating daily ZIP file in
user Download directory notifying RSM of
ready for download status

Download URL
User Check via AAA
Not OK
OK
User Download webpage at Data Centre with option
for downloading daily zip files option for
removing daily zip files
No access
If Removed
Status Downloaded
25
Request Status Manager and Download Manager

• RSM has Web service to communicate with the
  Download Managers, locally installed
• RSM keeps IP-addresses, EDMO codes, data manager
  e-mail, trigger addresses, generic download paths
  and V1/interim V1 labels of all connected Data
  Centres in config file
• Download Manager is Java programme, that must be
  locally installed
• Download Manager also has a config file with
  IP-address of RSM service, EDMO code, paths to
  support directories with data sets, database
  mapping files, and coupling table

26
Functionality Download Manager V1.1

• Communication with RSM Web service
• Retrieving and processing order lists after being
  triggered
• Retrieving data files, related to requested
  CDIs, via local_CDI_ID and local Coupling Table
• Support for retrieving data sets from file
  management systems and from database systems,
  following the data set requests and the
  corresponding indicators in the local Coupling
  Table
• If data sets are stored in database, then
  producing the data files in the agreed ODV format
  via read-method
• Note Data sets from a file management system
  must be pre-processed in SDN transport formats
  (ODV and optional MedATLAS for ASCII type data,
  or NetCDF for grid type data)
• Combining data set files in a new daily zip file
  or updating an existing daily zip file with the
  latest data sets and placing the daily zip files
  in a user download directory
• Generating dynamic user download webpage,
  following SDN template, to give user overview and
  option of downloadable zip files
• Note the download URL is composed of the generic
  download path of the Data Centre as specified in
  the RSM configuration file (at installation)
  the user login.
• Protecting the download URL via the AAA login
  service (CAS)
• Notifying the RSM Web service once a User has
  removed daily zip files (because the user has
  completed the download successfully) by
  indicating the status Downloaded for the
  associated data set requests
• Keeping a local log

27

Request Status Manager - Homepage
28

Submit Shopping list gt User Authentication
29

User RSM Overview of request per Data Centre
and Status
30

User RSM Status Ready for Download Download
option
31

User personal Download webpage at Data Centre
32

User RSM Status All
33
Monitoring Tracking

• All submitted requests, their users, the data
  sets concerned, and processing of these requests
  is administered in the Request Status Manager
• SeaDataNet Data centres can also log on to the
  Request Status Manager and oversee all requests,
  handle tbd requests and prepare a number of
  online reports.

34

Data centre RSM overview Standing Orders
35

Data centre RSM overview To be Discussed Orders
36

Data centre RSM overview History Orders
37
SeaDataNet URLs

• CDI V1 retrievalhttp//seadatanet.maris2.nl/v_cd
  i_v1/search.aspCDI V1 RSMhttp//seadatanet.mar
  is2.nl/request/welcome.html---------------------
  --------------------------------------------------
  ----
• CDI V1 RSM Web service  TESTERhttp//seadatanet
  .maris2.nl/test/rsmtest.aspxWSDL
  http//seadatanet.maris2.nl/rsm/RequestStatusManag
  er.asmx?wsdl

38
SeaDataNet V1 Data Access planning

• CDI V1 interface, database and XML import are
  ready
• Download Manager V1.1 ready
• Coming months all TTT partners will install the
  DM V1.1 and configure their infrastructure for
  being a full V1 data centre.
• Preparing CDI V1 XML records following the
  upgraded CDI XML schema and using the new MIKADO
  software
• Installing the Download Manager software
• Choosing between pre-processed data files OR
  conversion via the Download Manager
• Converting data files to SeaDataNet formats, in
  case of choice for pre-processed mode
• Operational by end 2008

39
SeaDataNet interim V1 Implementation by other
partners in 2008

• Coming months all other partners will also start
  populating the CDI V1 database
• Preparing CDI V1 XML records following the
  upgraded CDI XML schema and using the new MIKADO
  software
• Not yet installation of Download Manager, but
  data centres will be informed by the Request
  Status Manager by e-mail of all requests
• Data centres will process all requests manually
  using the Request Status Manager
• Effect is, that users will experience a common
  approach to all data centres. The difference is
  behind the scenes.
• In 2009 partners gradually moving to full V1
  status by installing DM and delivering data sets
  in SeaDataNet Formats (conversion via new NEMO
  tool).

40
Thank you