Online data access in the SeaDataNet V1 system

1

• Online data access in the SeaDataNet V1 system
• by
• Dick M.A. Schaap technical coordinator
• Oostende, June 08

2
SeaDataNet Infrastructure versions and planning

• Version 0 Continuation and maintenance of
  existing Sea-Search systems with minor
  modifications
• Version 1 All data centres involved in
  upgraded metadatabases and transparent data
  access. Prototype ready by March 2008 all TTT
  partners by end 2008 thereafter gradually
  migrating all partners operational by mid 2009.
• Version 2 All data centres involved in
  Transnational Access and Access upgraded. Plus
  viewing services and further virtualisation.
  Operational by 2010.

3
Discovery services

• EDMED - Data Sets
• EDMERP - Research Projects
• CSR - Cruise Summary Reports
• EDIOS - Monitoring systems
• EDMO - Marine Organisations
• CDI - Common Data Index

4
SeaDataNet Users portal schema V1
Data download
User registration
Status of request
Data request
General request
Metadata request
data
metadata
Project info
Registr.
Standards
Cross search
My transact.
Requestsstatus manager
CDI
Shopping basket
EDIOS
EDMED
Software
AAA
CSR
EDMERP
EDMO
Vocabularies
Organisation data source id
User Register
CSR
EDIOS
CDI
EDMO
BODC Database
EDMED
EDMERP
BSH Database
at BODC
at BSH
at MARIS
Metadata Data catalogues
...
Entry point for access hits
services for metadata
Data downloading services
Download managers in Data centers
Visualization services (WMS) for V2
5
Common Data Index CDI

• CDI is a fine-grained index to individual data
  measurements (such as a CTD cast or moored
  instrument record). This is in contrast to EDMED,
  which is an inventory of data sets, each of which
  comprises a large number of measurements.
• CDI paves the way for direct online data access
  through file downloads or online data requests.
• Present CDI V0 system gt 340.000 CDI entries
  from 36 Data Centres from 29 countries
• http//seadatanet.maris2.nl/v_cdi_v0/search.asp

6
Common Data Index CDI V0
7
Common Data Index CDI V0
8
Common Data Index CDI V0
9
CDI V0

• Central and homogeneous metadatabase, but for
  data access the users are confronted with the
  various user interfaces of the SeaDataNet data
  centres (which can provide online search and
  access / ordering facility / e-mail request)

User interface
CDI
url
url
url
Data centre User interface
Data centre User interface
Data centre User interface
data
data
data
10
CDI V1

• Central and homogeneous metadatabase plus
  transparent access to the distributed data sets
  via a unique user interface at the SeaDataNet
  portal

User interface
CDI
Central User Register
Shopping basket
Request Status Manager
Download Manager
Download Manager
Download Manager
data
data
data
11
CDI V1 System Modules

• CDI Metadatabase and User Interface
• Searching and browsing of metadata of data sets,
  managed by SeaDataNet Data Centres
• SeaDataNet Central User Register with User Roles
  db web service
• Details of users, their organizations and
  addresses, license period, Id-Passwords,
  SeaDataNet Roles of users
• SeaDataNet Shopping Basket
• Preparing a user request of multiple data sets,
  handling validation of users and roles via the
  Central User Register, and routing requests to
  the Request Status Manager
• SeaDataNet Request Status Manager
• Processing and administration of all requests
  and data deliveries (downloads), for users in
  communication with data centres. For data
  centres to oversee all transactions.

12
System Modules

• SeaDataNet Data centre Download Manager
• Handles communication with SeaDataNet Request
  Status Manager and takes care that requested
  files are made ready for downloading by User (if
  cleared! ) in the agreed formats via a local
  website address.

search shop
Data download
SeaDataNet Portal
Data centre
Data centre
Data centre
Data centre
Metadata transaction data
13
Order flow diagram
Search in CDI directory
CDI Meta database
Register Users Licenses

Results
Requests list
Shopping Basket
User Register
User Roles
Submit Requests (specify filetype? )
Log in
WS
User Check
Info about Conditions, Registration etc
NO
YES User Role Requests
14

CDI V1 Search home page
15

Search Results page
16

Search Detail page (full)
17

Search records selected for Shopping basket
18

Shopping basket
19

Submit Shopping list gt User Authentication
20
Authentication, Authorization and Administration

• Single Sign On system set up for access to
  distributed system
• Users authentication information based on
  personal login / password
• User must register to get one login, thereby
  agreeing with the SeaDataNet User Licence
• Central User Register, but management is
  decentralized
• Authorisation based on User Roles

21

User identified and request for motivation and
order confirmation
22

Request confirmed and link to RSM (also by e-mail)
23
YES User Role Requests

Shopping Basket
Submit extended Request Form with motivation
Requests user info in RSM Split by DC
Status Pending
Request Status Manager (RSM)
Confirmation email to User with RSM info
A B C D
U
V
W
Combi logic
Validation of access via business matrix

24
Data access restrictions in CDI V1
CC Collection cost charge
FE Commercial charge
LI Licence
MO Moratorium
NA No access
NC Distribution cost
OG Organisation
RS Restricted
SR Academic
UK Unknown
UN Unrestricted
See Vocabulary L081 for definitions Multiple
restrictions may apply
25
User Roles in AAA Register, of relevance for data
access
SDRN02 Public
SDRN03 Academic
SDRN04 Commercial
SDRN05 National and local government
SDRN06 Pan-National government

See Vocabulary C866 for definitions
Multiple roles may apply
26
Business Matrix
Restriction Role A B C D
U
V
W
TBD
TBD
NO
OK
OK
TBD
TBD
NO
NO
NO
OK
TBD
27
Request Status Manager (RSM)
Validation of access via business matrix
Data Centre
Download Manager

Status OK
Accepted requests in Download Manager
Status TBD
Status NO
E-mail to DC manager
Check in SDN User Register
Status OK
OK
Retrieving files preparing formats giving
URL for ZIP file
Not OK
Status Ready for Download
Validate in RSM
Access Denied
Status Access Denied
28
Request Status Manager (RSM)
Data Centre
Download Manager
Retrieving files preparing formats giving
URL for ZIP file

URL
Downloading data sets directly from Data Centres
Status Downloaded
29
Request Status Manager and Download Manager

• RSM has Web service to communicate with the
  Download Managers, locally installed
• RSM keeps IP-addresses, EDMO codes, data manager
  e-mail, trigger addresses and V0/V1 labels of all
  connected Data Centres in config file
• Download Manager is Java programme, that must be
  locally installed
• Functions DM
• Communication with RSM Web service
• Retrieving and processing order lists after being
  triggered
• Double checking user via AAA Web service
• Retrieving data files, related to requested
  CDIs, via local_CDI_ID
• Preparing data files in agreed and requested
  formats (ODV, MedATLAS, NetCDF)
• Placing data files in zip format for downloading
• Informing RSM Web service of order processing
  status and urls
• Keeping a local log
• First version DM slim version gt no NetCDF and
  use of pre-processed files
• Second version DM full version gt includes
  conversion of file formats and support for file
  management and database storage

30

Request Status Manager - Homepage
31

Submit Shopping list gt User Authentication
32

User RSM Overview of request per Data Centre
and Status
33

User RSM Status Ready for Download Download
option
34

User RSM Status All
35
Monitoring Tracking

• All submitted requests, their users, the data
  sets concerned, and processing of these requests
  is administered in the Request Status Manager
• SeaDataNet Data centres can also log on to the
  Request Status Manager and oversee all requests,
  handle tbd requests and prepare a number of
  online reports.

36

Data centre RSM overview Standing Orders
37

Data centre RSM overview To be Discussed Orders
38

Data centre RSM overview History Orders
39
SeaDataNet URLs

• CDI V1 retrievalhttp//seadatanet.maris2.nl/v_cd
  i_v1/search.aspCDI V1 RSMhttp//seadatanet.mar
  is2.nl/request/welcome.html---------------------
  --------------------------------------------------
  ----
• CDI V1 RSM Web service  TESTERhttp//seadatanet
  .maris2.nl/test/rsmtest.aspxWSDL
  http//seadatanet.maris2.nl/rsm/RequestStatusManag
  er.asmx?wsdl

40
SeaDataNet V1 Implementation planning

• Prototype is ready
• It includes a temporary set of CDI records and
  data sets, that are managed by RNODC
• Majority of CDI test records is unrestricted,
  but some records have other data access
  restrictions
• First version Download Manager installed gt ODV
  ASCII files and use of pre-processed files
• See http//seadatanet.maris2.nl/v_cdi_v1/search.as
  p
• Use your SeaDataNet login - password

41
SeaDataNet V1 Full implementation by TTT
partners in 2008

• Coming months all TTT partners will start
  installing the Download Manager and configuring
  their infrastructure for being a full V1 data
  centre.
• Preparing CDI V1 XML records following the
  upgraded CDI XML schema and using the new MIKADO
  software
• CDI must cover all available data sets
• Special attention to the data access
  restriction
• Installing the Download Manager software
• Filling the coupling table between the
  LOCAL_CDI_ID (in the CDI XML records) and the
  local Data sets
• Choosing between pre-processed data files OR
  conversion via the Download Manager
• Note data files can come from a local file
  management system OR from a local database via
  specific queries
• Must be ready end 2008

42
SeaDataNet interim V1 Implementation by other
partners in 2008

• Coming months all other partners will also start
  populating the CDI V1 database
• Preparing CDI V1 XML records following the
  upgraded CDI XML schema and using the new MIKADO
  software
• CDI must cover all available data sets
• Special attention to the data access
  restriction and LOCAL_CDI_ID
• Not yet installation of Download Manager, but
  data centres will be informed by the Request
  Status Manager by e-mail of all requests
• Data centres will process all requests manually
  using the Request Status Manager and indicating
  the URL where the user can download the data sets
• Must be ready end 2008
• Effect is, that users will experience a common
  approach to all data centres. The difference is
  behind the scenes.

43
Request Status Manager (RSM)
Validation of access via business matrix

Status OK
Status TBD
Data Centre
Status NO
E-mail to DC manager
Retrieving files giving URL for ZIP file
Status OK
Validate in RSM
Status Ready for Download
Access Denied
Status Access Denied
interim V1
44
Request Status Manager (RSM)
Data Centre
Retrieving files giving URL for ZIP file

URL
Downloading data sets directly from Data Centres
Check by DC Manager
Status Downloaded
interim V1
45
SeaDataNet full V1 Implementation by other
partners in first half of 2009

• Early 2009 all other partners will start with
  installation and configuration of the Download
  Manager, using the experience from the TTT
  partners
• Installing the Download Manager software
• Filling the coupling table between the
  LOCAL_CDI_ID (in the CDI XML records) and the
  local Data sets
• Choosing between pre-processed data files OR
  conversion via the Download Manager
• Note data files can come from a local file
  management system OR from a local database via
  specific queries
• Must be ready mid 2009

46
SeaDataNet V1 Demonstration

• See http//seadatanet.maris2.nl/v_cdi_v1/search.as
  p